Sensing Depth with 3D Computer Vision - Dr. Benjamin Busam
Our world is 3D. However, most computer vision sensing happens on a 2D pixel grid. To reason about distances, one needs to understand how far object pixels are in the real world where distance estimation from sensor input provides an essential tool for computer vision applications. For an autonomous vehicle to drive, it is essential to know the distance to cars, pedastrians and obstacles to not cause accidents; and for augmented reality applications to look convincing, 3D understanding of the scene is key to realistically hide augmentations behind scene content.
In this talk, we will give a background on different depth sensing technologies such as multi-view stereo, time-of-flight sensing, and monocular approaches - and look how sensor fusion, synthetic data and self-supervision can boost the performance of depth estimation. We identify major obstacles and present first proposals to overcome some of them.
00:00 Intro
03:42 Sensing Depth with 3DCV
05:02 Image is a 2D projection of a 3D scene
08:09 Mono Stereo Time Modalities
12:19 3D Sensors
15:28 Active Stereo Vision
16:35 Spatial Cues
18:26 Epipolar Geometry
21:08 Rectification
26:03 Image to World Stereo Triangulation
28:36 Dense Disparity Maps
29:46 Learning Stereo Vision
31:18 Vanilla Depth from Stereo
36:12 Structure is Important for the Matching Problem
39:13 Learning to Estimate Depth from RGB
40:57 Depth from a single image [ monocular depth ]
42:17 Mono Depth - Supervised
46:33 Mono Depth Self-Supervision with Stereo
53:00 Temporal Cues
53:28 Consistent Video Depth Estimation
56:34 TCD Attention meets Geometry
01:01:28 Sensor Fusion
01:04:36 ToF RGB Fusion Coarse Initial Alignment
01:05:38 ToF + RGB Fusion Strategy
01:08:16 Depth Completion Synthetic Data for RGB + LIDAR
01:10:25 Radar-based Depth Improvements
01:12:47 Open Research Problems
01:14:57 Sparse radar Signal for Dynamic Objects
01:16:39 Photometric Challenges for RGB-D
01:18:00 Depth Sensor Artefacts
01:21:20 GT Quality matters for Generalization
01:23:04 Discussion
[Chapters were auto-generated using our proprietary software - contact us if you are interested in access to the software]
Lecture slides: https://www.dropbox.com/s/j67kq0jqwsp...
The talk will take a tour through the literature in the field of depth estimation and exemplify some tricks and solutions with ideas of recent papers by the speaker such as:
Lopez-Rodriguez, Busam, Mikolajczyk. ACCV 2020.
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data
https://github.com/alopezgit/project-...
Jung, Brasch, Leonardis, Navab, Busam. 3DV 2021.
Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments
https://arxiv.org/abs/2112.03750
Gasperini, Koch, Dallabetta, Navab, Busam, Tombari. 3DV 2021.
R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes
https://arxiv.org/abs/2108.04814
Ruhkamp, Gao, Chen, Navab, Busam. 3DV 2021.
Attention meets Geometry: Geometry Guided Spatial-Temporal Attention for Consistent Self-Supervised Monocular Depth Estimation
https://github.com/DaoyiG/TC-Depth
Curriculum Vitae:
Benjamin Busam is a Senior Research Scientist with the Technical University of Munich coordinating the Computer Vision activities at the Chair for Computer Aided Medical Procedures. Formerly Head of Research at FRAMOS Imaging Systems, he led the 3D Computer Vision Team at Huawei Research, London from 2018 to 2020. Benjamin studied Mathematics at TUM. In his subsequent postgraduate programme, he continued in Mathematics and Physics at ParisTech, France and at the University of Melbourne, Australia, before he graduated with distinction at TU Munich in 2014. In continuation to a mathematical focus on projective geometry and 3D point cloud matching, he now works on 2D/3D computer vision for pose estimation, depth mapping and mobile AR as well as multi-modal sensor fusion and collaborative robotics. For his work on adaptable high-resolution real-time stereo tracking he received the EMVA Young Professional Award 2015 from the European Machine Vision Association and was awarded Innovation Pioneer of the Year 2019 by Noah's Ark Laboratory, London. Benjamin is part of the programme commitee for CVPR, ICCV, and ECCV, and was recently awarded with the 3DV Outstanding Reviewer Award consecutively in 2020 and 2021.
https://www.in.tum.de/campar/members/...
-------------------------
Find us at:
Newsletter for updates about more events ➜ http://eepurl.com/gJ1t-D
Sub-reddit for discussions ➜ / 2d3dai
Discord server for, well, discord ➜ / discord
Blog ➜ https://2d3d.ai