What are Mixture-of-Experts Models | ft. Aritra @HuggingFace
What are Mixture-of-Experts Models | ft. Aritra  @HuggingFace
Uploaded April 2026 | Updated November 2026, 21 hours ago
In this clip, Aritra Roy Gosthipaty from the Hugging Face Transformers team breaks down one of the most important (and often misunderstood) architectures in modern AI: Mixture-of-Experts models.

Main MOE explainer: what they are, why they became mainstream, and why the ecosystem shifted around them.

Chapters:
- 00:00 Why Mixture-of-Experts Models Matter
- 00:14 Mixture-of-Experts Layers
- 01:07 vLLM and Serving Stacks
- 01:51 DeepSeek-V2
- 02:55 Mixtral 8x7B
- 03:20 Switch Transformers
- 04:25 Inference Providers
- 05:12 Unsloth Kernels

Topics covered:
- Mixture-of-Experts Layers
- vLLM and Serving Stacks
- DeepSeek-V2
- Mixtral 8x7B
- Switch Transformers
- Inference Providers

Sources mentioned:
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — arxiv.org/abs/1701.06538
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — arxiv.org/abs/2309.06180
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — arxiv.org/abs/2405.04434
- Mixtral of Experts — arxiv.org/abs/2401.04088
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — arxiv.org/abs/2101.03961
- Inference Providers — huggingface.co/docs/inference-providers/index
- Unsloth Docs — unsloth.ai/docs

Listen to the full podcast on Spotify: open.spotify.com/show/2BWAr3zLa2xhUqoHlg8DAD?si=-nXiwfyyQfaowCqb58Ig-w

Watch the full conversation on YouTube: youtu.be/O3Ul6H20pLI
What are Mixture-of-Experts Models | ft. AritraHugging Face Journal Club: Scaling Laws for Pre-training & RLHow One API Runs 27 ML TasksHugging Face HackathonHow to make your very FIRST open-source contribution (feat. Gradio)From AI Agents to Faster Kernels: Ben Burtenshaw & Felix LeClair (AI Plumbers #2)Hugging Face Journal Club: Embarrassingly Simple Self-Distillation Improves Code GenerationHugging Face Journal Club: AsyncOPD and How Stale Can On-Policy Distillation Be?How to use Hugging Face models on VS Code CopilotTD-MPC Explained, With Alexander Soare (Part 2 of 2)Live workshop with HF co-founder and CEO: Step up your AI builder game with Enterprise HubDive deep with The MCP Course! Q&A on unit 1 of the MCP Course
Hugging Face |

What are Mixture-of-Experts Models | ft. Aritra

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER