Video-to-video translation, video object tracking, video super-resolution using CoDeF
Content deformation field (CoDeF) is a new type of video representation consisting of two main parts each consisting of Multi-resolution hash encoding and a small MLP. Part 1 is a 2D hash based canonical content field aggregating the static contents in the entire video. Part 2 is a 3D hash based temporal deformation field recording the transformations from the canonical image to each individual frame. Such a new type of representation naturally supports lifting image algorithms for video processing, in the way of directly applying the established algorithm on the canonical image (i.e., rendered from the canonical content field) and then propagating the results along the time axis through the temporal deformation field.
In this video, I will talk about the following: What are video generation challenges and how does CoDeF handle them? What are content deformation fields? How is CoDeF model trained? Downstream video processing applications for CoDeF like Video-to-video translation, Video object tracking, Video super resolution, User interactive video editing.
For more details, please look at https://arxiv.org/pdf/2308.07926.pdf
Ouyang, Hao, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. "Codef: Content deformation fields for temporally consistent video processing." arXiv preprint arXiv:2308.07926 (2023).