Apple's MM1 Paper Breaks Down Their Groundbreaking LLM Training Methods
Foundation models that you can use are of two types. They are either closed models such as GPT4. You can use them but we know almost nothing about them. Or they are Open models in which case they release detailed descriptions of data, model architecture and training configuration. In this case, there is almost no information about the process which resulted in this exact model.
Apple has done something which you will never expect out of them. They have released this paper with detailed descriptions of the process that they followed when creating the MM1 paper. This will be a great resource for teams which are working on training Multimodal Large Language Models.
⏱️ Timestamps
0:00 Intro
1:04 Abstract
3:04 Description of the Model
6:19 Mixture of Experts
9:07 Recipe for Building MM1
14:31 Model Architecture Ablations
20:21 Data Ablations
22:08 Final Model and Training Recipe
26:13 Supervised Fine-Tuning
27:38 Final Details and Some Examples
🔗 Links
MOE huggingface: https://huggingface.co/blog/moe
VIT Large: https://huggingface.co/openai/clip-vi...
CLIP model: https://towardsdatascience.com/clip-m...
Honeybee paper: https://arxiv.org/pdf/2312.06742.pdf
Contrastive loss explained: https://towardsdatascience.com/contra...
Autoregressive Image Modeling: https://uvadlc-notebooks.readthedocs....
Axlearn framework: https://github.com/apple/axlearn
LLAVA Next: https://huggingface.co/docs/transform...
LLAVA Explained: https://encord.com/blog/llava-large-l...
🔗 Career growth
Career Guidance in Machine Learning: https://topmate.io/joydeep_bhattacharjee
FREE Mock machine learning interview coach: https://vibrantai.academy/interview-t...
NLP basics: https://vibrantai.academy/courses/1/
Connect on LinkedIn: / joydeep-bhattacharjee-934a1157
Follow me on X: / alt227joydeep
👋🏻 About Me
My name is Joydeep Bhattacharjee and I talk about GenAI, career and AI industry. Reach out to me: topmate.io/joydeep_bhattacharjee
#llm #apple #multimodal #largelanguagemodels #finetuning #transformers