Open3DSG: an open-vocabulary 3D scene graph generation method
With #CVPR2024 coming soon, check out the conversation of Harpreet Sahota 🥑’s (Hacker-in-Residence- Voxel51) and Jason Corso (Co-Founder and Chief Science Officer - Voxel51) with Sebastian Koch PhD Student - Bosch Research.
We discussed Sebastian's work, Open3DSG, an open-vocabulary 3D scene graph generation method that predicts scene graphs from point clouds.
Unlike traditional methods, Open3DSG uses vision language models to identify and describe objects and their relationships in a zero-shot manner.
🤔 How Open3DSG Works
Open3DSG integrates features from large multimodal language models into the 3D domain. Previously, models like BLIP and CLIP were used in 2D.
Sebastian's team separated the link between the image encoder and the language decoder, allowing them to transfer information from 2D vision language models into their 3D network.
💬 "We want to learn this graph neural network, which takes in the point cloud and then uses a graph neural network to produce a feature representation in graph form. These features in the graph exist in this vision language space and can be queried for relationships," - Sebastian.
👨💻 Applications
Open3DSG's standout feature is its ability to provide more detailed relationships between objects. For example, instead of stating that a TV is "on" the wall, it can specify that the TV is "mounted on" the wall—a crucial distinction for practical applications like robotic interaction.
💬 "Predicting open vocabulary scene graphs allows for more expressive relationships, which are crucial for tasks like robotic manipulation," says Sebastian.
The system turns the scene into a visual representation and then uses a language model to suggest and determine connections. These connections are not limited to predefined categories but include detailed explanations and potential uses, making the model more versatile.
🌁 Addressing Semantic Segmentation
They use Mask3D to preprocess point clouds. This segmentation method is adept at grouping points into instances, which is fundamental for understanding the object relationships in a scene.
💬 "We use a method called mask3D for predicting class-agnostic masks of the point cloud. This helps us to group points in a shared instance," emphasizes Sebastian.
⚒ Practical Use
Sebastian encourages practitioners to experiment with their existing
Open3DSG checkpoint across different domains. The model's open vocabulary allows it to transfer across various scenarios without extensive retraining, ensuring broader applicability.
The Future of Open Vocabulary 3D Scene Graphs
Open3DSG enables machine understanding of complex environments, with applications in robotics and indoor scene analyses.
💬"We believe our method's open vocabulary and adaptability will be particularly impactful in indoor environments and for robots requiring detailed relational understanding," concludes Sebastian.