NExT-GPT: Any-to-Any Multimodal LLM
In this video we explain NExT-GPT, a multimodal large language model (MM-LLM), that was introduced in a research paper titled: "NExT-GPT: Any-to-Any Multimodal LLM".
We carefully review the NExT-GPT framework, explaining its different components, to understand how it is capable of using a LLM as its core agent to both process input and generate output from multiple modalities.
We then review a multimodal conversation example to get a better intuition for what can be done with such a framework.
Next, we dive into how NExT-GPT was trained by explaining few diagrams from the paper.
Finally, we review interesting results from the paper.
The multimodal input encoders used by NExT-GPT are from ImageBind, a multimodal model by Meta AI which we've covered in the following video - • ImageBind from Meta AI - One Embeddin...
We also explain this paper here - https://aipapersacademy.com/next-gpt/
Arxiv page - https://arxiv.org/abs/2309.05519
Project page - https://next-gpt.github.io/
👍 Please like & subscribe if you enjoy this content
-----------------------------------------------------------------------------------------------------
Support us - https://paypal.me/aipapersacademy
We use VideoScribe to edit our videos - https://tidd.ly/44TZEiX (affiliate)
We use ChatPDF to analyze research papers - https://www.chatpdf.com/?via=ai-papers (affiliate)
-----------------------------------------------------------------------------------------------------
Chapters:
0:00 Introduction & Motivation
1:03 NExT-GPT Framework
4:36 Conversation Example
5:32 Training NExT-GPT
8:40 Results