Validating Generative AI with Simulation for Industrial Robots
Many tasks, like machine tending, are very well understood, yet still take considerable time to program correctly. This makes them a perfect candidate for using Large Language Models (LLM) such as OpenAI's GPT-4 to create the programs for these tasks. The challenge with this is how to make sure that the program generated by the LLM actually works properly. Generative AI is still very unpredictable in its output, so how can we reliably use this for something as sensitive as industrial manufacturing?
Using NVIDIA Omniverse, we can provide annotations about a scene so the LLM has concrete goals to generate the program from, such as where workpieces need to be placed in the environment. This provides a semantically grounded prompt and a program that is relevant to the situation. Then, we can load the program into READY Robotics ForgeOS and run the program directly in the same simulation. By validating in sim, we can be sure the program works and doesn't have any unforeseen behavior. Then using ForgeOS, we can seamlessly transfer the generated program to the physical robot workcell and run it there.