With OpenAI, Figure 01 robot 🤖 can now have full conversations with people • subscribe✨ 👏
Revolutionizing AI Interactions: An In-Depth Look at Figure 01's Conversational Capabilities with OpenAI
With OpenAI, Figure 01 can now have full conversations with people
-OpenAI models provide high-level visual and language intelligence
-Figure neural networks deliver fast, low-level, dexterous robot actions
Everything in this video is a neural network: Let's break down what we see in the video:
All behaviors are learned (not teleoperated) and run at normal speed (1.0x).
We feed images from the robot's cameras and transcribed text from speech captured by onboard microphones to a large multimodal model trained by OpenAI that understands both images and text.
The model processes the entire history of the conversation, including past images, to come up with language responses, which are spoken back to the human via text-to-speech. The same model is responsible for deciding which learned, closed-loop behavior to run on the robot to fulfill a given command, loading particular neural network weights onto the GPU and executing a policy.
Connecting Figure 01 to a large pretrained multimodal model gives it some interesting new capabilities. Figure 01 + OpenAI can now:
Describe its surroundings.
Use common sense reasoning when making decisions. For example, "The dishes on the table like that plate and cup are likely to go into the drying rack next".
Translate ambiguous, high-level requests like "I'm hungry" to some context-appropriate behavior like "hand the person an apple".
Describe why it executed a particular action in plain english. For example, "It was the only edible item I could provide you with from the table".
A large pretrained model that understands conversation history gives Figure 01 a powerful short-term memory.
Consider the question, "Can you put them there?" What does "them" refer to, and where is "there"? Answering correctly requires the ability to reflect on memory.
With a pretrained model analyzing the conversation's image and text history, Figure 01 quickly forms and carries out a plan: 1) place the cup on the drying rack, 2) place the plate on the drying rack.
Finally, let's talk about the learned low-level bimanual manipulation.
All behaviors are driven by neural network visuomotor transformer policies, mapping pixels directly to actions. These networks take in onboard images at 10hz, and generate 24-DOF actions (wrist poses and finger joint angles) at 200hz.
These actions serve as high-rate "setpoints" for the even higher rate whole body controller to track.
This is a useful separation of concerns:
Internet-pretrained models do common sense reasoning over images and text to come up with a high-level plan.
Learned visuomotor policies execute the plan, performing fast, reactive behaviors that are hard to specify manually, like manipulating a deformable bag in any position.
Meanwhile, a whole body controller ensures safe, stable dynamics. For example, maintaining balance.
Even just a few years ago, I would have thought having a full conversation with a humanoid robot while it plans and carries out its own fully learned behaviors would be something we would have to wait decades to see. Obviously, a lot has changed :).
In my mind, Figure is the best place in the world right now to scale up embodied AI.
#figure01robot #openai #robotics #ai #technology