Can OpenAI Sora Solve 3D Modeling? I Tried This...
Can we turn AI videos into 3D models? Can OpenAI Sora solve the problem of text to 3D? In this video, I'll show you how to go from video to 3d using Gaussian Splatting and share my thoughts as a machine learning researcher.
The demos in this video are Gaussian splats and can be converted to meshes via tools.
-
This video discusses the potential of converting AI-generated videos into 3D models, with a focus on using OpenAI's Sora video generation model and Gaussian splatting techniques. Here's a summary of the key points:
Cindy demonstrates using Polycam's Gaussian Splatting feature to convert Sora-generated videos into 3D models. Results vary depending on the video content, with arc shots and rotating camera movements producing better 3D reconstructions than tracking shots. Factors affecting 3D conversion quality include:
3D consistency of the video
Minimal subject movement
Limitations of the conversion algorithm
While current results are not ideal for applications like game development, which requires clear object topologies, the technology shows promise for creating 3D scenes for movies and other media.
The video-to-3D approach could potentially allow directors to generate virtual spaces for filming using AI.
Industry trends suggest a growing connection between video and 3D generation, with companies like Luma and Gemo AI showing interest in both fields.
Cindy believes that as algorithms improve and category-specific treatments are developed, the quality of AI-generated 3D models from videos will significantly improve.
While text-to-video models like Sora don't currently solve all text-to-3D challenges, they represent a significant step forward in generating 3D scenes from scratch without scanning real objects.
-
Chapters:
0:00 Intro
0:16 Polycam
0:30 Tacking Shots
0:50 Arc Shots
1:23 Algorithm
1:45 Quality Factors
2:50 Potential Improvements
3:27 Scenes vs. Objects
3:56 Game Industry
4:34 Startup Trends
5:15 Ending