Virtual Production with Blackmagic Studio Camera 4K Pro
Trying out some new techniques for #virtualProduction in Unity for Games. Here I'm using a Blackmagic Studio 4k Pro camera and projecting a virtual background that gets tracked in realtime with a help of a single april-tag. It's a simple solution for a complex problem of syncing external trackers (like VIVE). In this case I don't have to worry about latency issues, as all processing is done on image based recognition. I can also use mask objects to mask any trackers and areas that aren't covered by greenscreen, while still having an ability to pan my camera freely in real world.
To add to this, I'm able to preview all of this on my Blackmagic camera screen.
Pros of this approach:
-You can do some things that otherwise require a hardware trackers that are hard to sync with camera. Image tags are a good way to do this almost for free. You can be sure that your camera is always synced with your mask objects/background.
-It could be used to enable pixel perfect tracking with trackers like VIVE by doing some fusion of tracking data.
Cons:
-This approach for now works stable on a panning camera (on tripod). It mostly calculates rotation of a camera.
-It still isn't optimized for spatial positioning. (but it could be in a future)
-Depending on a shutter speed, fast camera moves produce a motion blur. This means that with fast movements, it isn't possible to track their motion. Option is to use faster shutter speeds and implement alternative tracking (like VIVE trackers).
-Tracker can go out of view. So I will probably need more of trackers and some solutions for masking them. Right now I'm using white paper to print them. Then I'm using green illumination to light a greenscreen. This makes trackers green. I will have to experiment with different colors and see what color can be keyed out while keeping good contrast for tracking.
-This setup needs a careful trackers position, and measuring a distance from a camera.
My main concern was to have precise rotational estimation as this is one of the most critical passes on video tracking. And a problem increases when you zoom more into an image. Wide angle shots aren't a big problem with external trackers.