Using Gemini AI to Control a 3D Scene in Unreal Engine
A real-time 3D configurator prototype built in Unreal Engine that you control entirely through natural language, using Gemini.
I was tired of clicking through endless menus and options in traditional configurators, so I explored a different approach. This project lets a user connect to the experience with their phone and simply chat with an AI assistant. You can ask it to change the car's color, alter the weather and time of day, or even change the environment to match your needs.
One of my favorite features is the ability to ask the AI to take a high-resolution screenshot of your creation at any moment, which is then sent directly back to your phone to save or share.
Dive deeper into the project and read the full technical case study on my website:
► Case Study: https://www.nilsbakker.nl/lab/ai-conf...
► My Portfolio: www.nilsbakker.nl
Technical Background:
This experience is built in Unreal Engine 5 and runs a WebSocket server. A simple web app on a phone connects to it, sending user messages to the Google Gemini AI for processing. The AI interprets the request and sends back a structured JSON command via the WebSocket to Unreal Engine, which then executes the changes in real-time using Blueprints.
#UnrealEngine #AI #GameDev #Realtime3D #TechnicalArtist #Innovation #UX #MadeWithUnreal