I trained a SmolVLA in Unity
🔗 GitHub Repository with all scripts: https://github.com/Grigorij-Dudnik/Sh...
🔗 My Unity project: https://drive.google.com/file/d/1gv0x... (Unity version - 6000.1.14f1)
🔗 Free Robot Arm Asset Used in the video: https://assetstore.unity.com/packages...
🔗 Clean Coder - AI coder I used: https://github.com/Grigorij-Dudnik/Cl...
Curious if you can train a powerful robot model like SmolVLA inside a Unity game engine? In this comprehensive Unity tutorial, I show you exactly how to connect the LeRobot framework with a custom Unity robotics environment to create your own embodied AI agent. This video is a deep dive into the entire process, covering the struggles and the solutions. You will learn how to set up your own robot simulation, implement an imitation learning strategy to collect a high-quality training dataset, and bridge the gap between Python and Unity for real-time inference. If you've been hitting a wall trying to use a VLA model with your own custom environment, this guide is for you. All the code, including the Unity scripts and the Python converter for the LeRobot dataset format, is available on my GitHub.
TIMESTAMPS:
0:00 - The Challenge: Training SmolVLA in Unity
0:50 - Why Choose Unity for Robot Simulation? (vs. ROS, Isaac Sim, Webots)
2:57 - Building the Simulator & Teacher for Imitation Learning
5:02 - Preparing a dataset
7:04 - Training Your VLA Model with LeRobot
7:35 - Bridging the Gap: Connecting Python & Unity for Inference
8:11 - First Results & The Refinement Loop
9:25 - Pushing the Limits: A Multi-Task Learning Test
9:58 - Final Verdict: SmolVLA and Unity
In this guide to AI robotics, we explore why Unity can be a good choice for your robot arm simulation projects, offering flexibility that other simulators can't match. I'll share the critical script that converts your data into the LeRobot format, saving you days of frustration. Finally, we'll train the SmolVLA model and run it live in our Unity robotics scene. We'll also examine the model's limitations, particularly with multi-task learning, and discuss what this means for using a VLA model in your DIY robot projects. This video is your complete roadmap to creating and training an embodied AI agent from scratch.