Uploaded July 2024 | Updated September 2026, 2 weeks ago

---

To try everything Brilliant has to offer—free—for a full 30 days, visit brilliant.org/bycloud . You’ll also get 20% off an annual premium subscription.

Mixture of Experts explained, well, re-explained. We are in the Fine-Grain era of Mixture of Experts and it's about to get even more interesting as we further scale it up.

This video was sponsored by Brilliant

Check out my newsletter:
mail.bycloud.ai

Special thanks to LDJ for helping me with this video

Mixtral 8x7B Paper
[Paper] arxiv.org/abs/2401.04088

Sparse MoE (2017)
[Paper] arxiv.org/abs/1701.06538

Adaptive Mixtures of Local Experts (1991)
[Paper] https://direct.mit.edu/neco/article-abstract/3/1/79/5560/Adaptive-Mixtures-of-Local-Experts?redirectedFrom=fulltext

Gshard
[Paper] arxiv.org/pdf/2006.16668

Branch-Train Mix
[Paper] arxiv.org/pdf/2403.07816

DeepSeek-MoE
[Paper] arxiv.org/abs/2401.06066

MoWE (from the meme at 7:51)
[Paper] arxiv.org/abs/2311.10768

Mixture of A Million Experts
[Paper] web3.arxiv.org/abs/2407.04153


This video is supported by the kind Patrons & YouTube Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Robert Zawiasa, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon, Claxvii 177th, Inferencer, Michael Brenner, Akkusativ, Oleg Wock, FantomBloth

[Discord] discord.gg/NhJZGtH
[Twitter] twitter.com/bycloudai
[Patreon] patreon.com/bycloud

[Music] massobeats - daydream
[Profile & Banner Art] twitter.com/pygm7
[Video Editor] @Askejm