Distinguished AI Lecture Series | Differentiable Optics and ISPs for End-to-end Camera Design
The Department of Computing, Imperial College London Presents...
Jean-François Lalonde, and his lecture: Differentiable Compound Optics and Black-box Image Processing for End-to-end Camera Design
Most modern commodity imaging systems we use directly for photography---or indirectly rely on for downstream applications---employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today's compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, i.e., deviations from Gauss' linear model of optics, instead of application specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly-parameterized neural network processing. This is complicated by the fact that configuration parameters of black-box ISPs often have complex interactions with the output image, and must be adjusted prior to deployment according to application-specific quality and performance metrics. Today, this search is commonly performed manually by "golden eye" experts or algorithm developers leveraging domain expertise, a process which is not compatible with end-to-end joint optimization.
In this talk, I will present optimization methods for modelling compound optics as well as hardware ISPs that lift these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and with application-specific end-to-end losses. To this end, we propose a learned, differentiable forward model for compound optics as well as for hardware ISPs, and an alternating proximal optimization method that handles function compositions with highly-varying parameter dimensions for optics, hardware ISP and neural nets. We assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting---together with hardware ISP post processing and detection---outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across all domain-specific applications tested.
About Jean-François Lalonde..
Jean-François Lalonde, Ph.D., is an Associate Professor in the Electrical and Computer Engineering Department at Université Laval since 2013. Previously, he was a Post-Doctoral Associate at Disney Research, Pittsburgh. He received a Ph.D. in Robotics from Carnegie Mellon University in 2011. His Ph.D. thesis won the CMU School of Computer Science Distinguished Dissertation Award. His research interests lie at the intersection of computer vision, computer graphics, and machine learning. In particular, he is interested in exploring how physics-based models and data-driven machine learning techniques can be unified to better understand, model, interpret, and recreate the richness of our visual world. To this end, his group has captured and published the largest datasets of indoor and outdoor high dynamic range wide-angle and omnidirectional images, freely available for research. He is actively involved in bringing research ideas to commercial products, as demonstrated by his patents and technology transfers with large companies such as Adobe and Facebook, and involvement as scientific advisor for high tech start-ups.
More info at http://vision.gel.ulaval.ca/~jflalonde/