Mixture of Experts (MoE), Visually Explained
The Mixture of Experts (MoE) architecture underpins many of today’s most advanced AI models, enabling massive increases in model parameters without compromising training or inference speed.
But how does it work? 🤔
In this video, let’s learn the key ideas behind MoE from the first principles.
Chapters:
00:00 Introduction
00:19 A transformer Block
00:54 Normalization
01:41 The role of Feedforward Networks (FFN)
05:53 Mixture of Experts (MoE)
07:22 Router Design
09:37 Fine-grained Experts
10:29 Shared Experts
11:41 Token-to-Expert Assignments
17:26 Load Balancing with Auxiliary Loss
21:29 Simplified Load Balancing Loss
24:31 Auxiliary-Loss-Free Load Balancing
26:54 Stabilizing training with Router Z-loss
31:20 Conclusion
References:
• [Sparsely-Gated MoE] https://arxiv.org/abs/1701.06538
• [GShard] https://arxiv.org/abs/2006.16668
• [Switch Transformer] https://arxiv.org/abs/2101.03961
• [Dropless MoE] https://arxiv.org/abs/2211.15841
• [Router Z-loss] https://arxiv.org/abs/2202.08906
• [DeepSeek MoE] https://arxiv.org/abs/2401.06066
• [DeepSeek V3] https://arxiv.org/abs/2412.19437
• [OLMoE] https://arxiv.org/abs/2409.02060
Additional resources:
• Mixture of Experts Lecture from Stanford University: • Stanford CS336 Language Modeling from Scra...
• A Visual Guide to Mixture of Experts (MoE) in LLMs: • A Visual Guide to Mixture of Experts (MoE)...
• How might LLMs store facts: • How might LLMs store facts | Deep Learning...
(Re-uploading the video due to rendering errors in the previous one.)
Video made with Manim https://www.manim.community/