1 Million Tiny Experts in an AI? Fine-Grained MoE Explained
To try everything Brilliant has to offer—free—for a full 30 days, visit https://brilliant.org/bycloud/ . You’ll also get 20% off an annual premium subscription.
Mixture of Experts explained, well, re-explained. We are in the Fine-Grain era of Mixture of Experts and it's about to get even more interesting as we further scale it up.
This video was sponsored by Brilliant
Check out my newsletter:
https://mail.bycloud.ai
Special thanks to LDJ for helping me with this video
Mixtral 8x7B Paper
[Paper] https://arxiv.org/abs/2401.04088
Sparse MoE (2017)
[Paper] https://arxiv.org/abs/1701.06538
Adaptive Mixtures of Local Experts (1991)
[Paper] https://direct.mit.edu/neco/article-a...
Gshard
[Paper] https://arxiv.org/pdf/2006.16668
Branch-Train Mix
[Paper] https://arxiv.org/pdf/2403.07816
DeepSeek-MoE
[Paper] https://arxiv.org/abs/2401.06066
MoWE (from the meme at 7:51)
[Paper] https://arxiv.org/abs/2311.10768
Mixture of A Million Experts
[Paper] https://web3.arxiv.org/abs/2407.04153
This video is supported by the kind Patrons & YouTube Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Robert Zawiasa, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon, Claxvii 177th, Inferencer, Michael Brenner, Akkusativ, Oleg Wock, FantomBloth
[Discord] / discord
[Twitter] / bycloudai
[Patreon] / bycloud
[Music] massobeats - daydream
[Profile & Banner Art] / pygm7
[Video Editor] @Askejm