Low-Latency 3D AI Characters in Unity with Real-Time Lip Sync | Convai Unity Tutorial
In this tutorial, we walk through how to build a fully interactive AI character in Unity using Convai’s updated WebRTC-based Unity plugin.
Starting with SDK installation through the Unity Package Manager or Unity Asset Store, this tutorial covers project setup, API key configuration, character and player setup, transcript UI integration, and real-time lip sync setup on a custom avatar.
We also show how the new Unity plugin enables lower response latency, hands-free conversation through voice activity detection, more contextual responses with long-term memory, and high-quality facial animation powered by Convai’s in-house NeuroSync model.
Feature Highlights:
Install Convai in Unity via UPM or Asset Store
Set up Convai Manager, Player, and Character components
Configure hands-free or push-to-talk interaction
Enable transcript UI for in-scene conversations
Add real-time lip sync and facial animation with NeuroSync
Use lower-latency WebRTC voice interaction
Build more contextual AI responses with long-term memory and knowledge integration
Integrate lifelike AI avatars for games, simulations, training, and immersive virtual experiences
This tutorial features Susan, an AI character used to demonstrate natural voice interaction in hands-free mode, accurate lip sync, and contextual conversations inside Unity.
Latest Unity Plugin Documentation: https://docs.convai.com/api-docs/plug...
UPM Installation Method: https://docs.convai.com/api-docs/plug...
UAS Installation Method: https://docs.convai.com/api-docs/plug...
Convai Unity Plugin: https://assetstore.unity.com/packages...
Check out our detailed blog on bringing interactive AI avatars to Unity projects:
https://convai.com/blog/unity-tutoria...
Time Stamps:
00:00 Demo
00:19 Intro
01:21 Unity Package Manager (UPM) Setup for Convai
01:54 Unity Sample Project Highlights
02:03 Unity Asset Store (UAS) Setup for Convai
02:57 Integrating Convai into Project
03:23 Setup Interaction Mode: Handsfree or Push-to-talk
03:32 API Key Setup
03:43 Convai Player Setup
03:57 Convai Character Setup
04:43 Adding Transcript UI
04:57 Testing without Lipsync
05:19 Organizing the Project
05:31 Lipsync Configuration
06:14 Final Test
08:37 Outro
Sign up at https://convai.com to create AI characters and integrate them across a variety of virtual world applications. And learn more about how Convai’s plugins can be used across the board.
Have questions or need assistance?
Visit the Convai Developer Forum – our dedicated community and support team are here to help with bug reports and technical queries. Get the support you need and connect with other developers to enhance your experience.
Convai Developer Forum: https://forum.convai.com/
Follow us on your favorite platforms to stay in touch:
LinkedIn: / convaitech
Reddit: / convai
Twitter: / convaitech
Instagram: / conv.ai
Facebook: / convaiofficial
TikTok: / conv.ai
ArtStation: https://www.artstation.com/convai