Learning 3D Representations from 2D Images
View synthesis has been a long-standing problem in the computer graphics and vision community. Finding an efficient and expressive representation is the key to addressing this problem.
In this talk, I will introduce two of our recent projects in this frontier. The first is Neural Light-transport Fields (NeLF), which enables simultaneous view synthesis and relighting from casual portrait photos.
The other work is Deep 3D Mask Volume, a way to enable flicker-free view synthesis for dynamic scenes. I will talk about how we design the representations and the underlying network to generate them.
Also, I will outline some possible research directions in this area.
00:00 Intro
10:58 Layered Representations Multiplane Images
20:04 How to Represent Our Visual World?
21:10 Deep Mask Volume for View Synthesis of Dynamic Scenes – ICCV 2021
27:00 Proposed 3D Mask Volume Approach
34:46 Dataset
37:16 Proposed 3D Mask Volume Approach
40:13 Dataset
45:42 Comparison to State-of-the-Art Binocular View Extrapolation Methods [Mildenhall et al. 2019]
47:14 Deep 3D Mask Volume for View Synthesis of Dynamic Scenes
48:50 Experimental Results
50:03 Conclusions
55:31 NeLF: Neural Light-transport Field for Synthesis and Relighting EGSR 2021 - Motivation
57:43 Method: Architecture
01:06:05 Method: Light-Transport
01:17:40 Method: Training Details
01:21:38 Results on Synthetic Dataset
01:25:47 Future Directions
[Chapters were auto-generated using our proprietary software - contact us if you are interested in access to the software]
Lecture slides: https://drive.google.com/file/d/1JZhy...
The talk is based on our recent papers:
(1) Deep 3D Mask Volume for View Synthesis of Dynamic Scenes, ICCV 2021
Project page: http://zhiqiangshen.com/projects/LS_a...
Git: https://github.com/ken2576/deep-3dmask
(2) NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting, EGSR 2021
Project page: https://cseweb.ucsd.edu//~viscomp/pro...
Git: https://github.com/ken2576/nelf
Presenter Bio:
Kai-En Lin is a 4th-year PhD student at UC San Diego advised by Prof. Ravi Ramamoorthi. Before that, he graduated with a bachelor degree in Electrical Engineering from National Taiwan University.
His research interests cover computer vision, image-based rendering and view synthesis. To be more specific, he focuses on how to represent the 3D visual world given a sparse set of 2D images.
He is a recipient of the Qualcomm FMA fellowship.
More information can be found at: https://cseweb.ucsd.edu/~k2lin/
-------------------------
Find us at:
Newsletter for updates about more events ➜ http://eepurl.com/gJ1t-D
Sub-reddit for discussions ➜ / 2d3dai
Discord server for, well, discord ➜ / discord
Blog ➜ https://2d3d.ai