How We Can Convert Any Videos to 3D
Videos are windows to another world. But the videos today are *flat*, confined to the original viewpoints. We showcase a method for converting any 2D videos into 3D videos that allow free-view synthesis.
Fast View Synthesis of Casual Videos
Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu
European Conference on Computer Vision, 2024
📝 Paper: https://arxiv.org/abs/2312.02135
🌐 Website: https://casual-fvs.github.io/
Abstract: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.
Transcript: Available in the subtitle.
Credit: The "versus" icon is from PNGtree.