Uploaded April 2026 | Updated November 2026, 3 days ago
In this episode, Alejandro sits down with Aritra Roy Gosthipaty from the Hugging Face Transformers team to talk about mixture-of-experts models, why dense models still matter, how synthetic data changes training, and what coding agents are changing for working engineers.
They discuss Mixtral, DeepSeek-V2, Switch Transformers, vLLM, Inference Providers, TinyAya, data curation, local inference limits, and how practitioners should think about coding with agents without losing core engineering skill.
More from Aritra:
- Hugging Face profile — huggingface.co/ariG23498
Chapters
00:00 Meet Aritra Roy Gosthipaty
00:45 How Aritra joined Hugging Face
03:05 What a Developer Advocate on Transformers works on
04:00 What mixture-of-experts models are
08:26 Why MOEs matter now
11:36 Where dense models still win
15:00 Where to start learning and training MOEs
18:44 Synthetic data, data engines, and data quality
22:18 Why MOEs are still hard to run locally
23:11 How coding tools changed engineering work
25:49 Do agents weaken creativity and skill?
28:29 Should beginners rely on coding agents?
33:21 What coding will look like in a year
35:25 The biggest recent AI wow moments
If you enjoyed the episode, subscribe for more conversations about open models, infrastructure, and the future of AI.
---
Sources / References
• Aritra Roy Gosthipaty — huggingface.co/ariG23498
• Mixture of Experts Explained — huggingface.co/blog/moe
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — arxiv.org/abs/1701.06538
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — arxiv.org/abs/2309.06180
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — arxiv.org/abs/2405.04434
• Mixtral of Experts — arxiv.org/abs/2401.04088
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — arxiv.org/abs/2101.03961
• Hugging Face Inference Providers — huggingface.co/docs/inference-providers/index
• Unsloth Docs — unsloth.ai/docs
• MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — arxiv.org/abs/2402.14905
• Tiny Aya: Bridging Scale and Multilingual Depth — arxiv.org/abs/2603.11510
• Aya 23: Open Weight Releases to Further Multilingual Progress — arxiv.org/abs/2405.15032
• Self-Instruct: Aligning Language Models with Self-Generated Instructions — arxiv.org/abs/2212.10560
• The Curse of Recursion: Training on Generated Data Makes Models Forget — arxiv.org/abs/2305.17493
• DeepSeek-V3 Technical Report — arxiv.org/abs/2412.19437
In this episode, Alejandro sits down with Aritra Roy Gosthipaty from the Hugging Face Transformers team to talk about mixture-of-experts models, why dense models still matter, how synthetic data changes training, and what coding agents are changing for working engineers.
They discuss Mixtral, DeepSeek-V2, Switch Transformers, vLLM, Inference Providers, TinyAya, data curation, local inference limits, and how practitioners should think about coding with agents without losing core engineering skill.
More from Aritra:
- Hugging Face profile — huggingface.co/ariG23498
Chapters
00:00 Meet Aritra Roy Gosthipaty
00:45 How Aritra joined Hugging Face
03:05 What a Developer Advocate on Transformers works on
04:00 What mixture-of-experts models are
08:26 Why MOEs matter now
11:36 Where dense models still win
15:00 Where to start learning and training MOEs
18:44 Synthetic data, data engines, and data quality
22:18 Why MOEs are still hard to run locally
23:11 How coding tools changed engineering work
25:49 Do agents weaken creativity and skill?
28:29 Should beginners rely on coding agents?
33:21 What coding will look like in a year
35:25 The biggest recent AI wow moments
If you enjoyed the episode, subscribe for more conversations about open models, infrastructure, and the future of AI.
---
Sources / References
• Aritra Roy Gosthipaty — huggingface.co/ariG23498
• Mixture of Experts Explained — huggingface.co/blog/moe
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — arxiv.org/abs/1701.06538
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — arxiv.org/abs/2309.06180
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — arxiv.org/abs/2405.04434
• Mixtral of Experts — arxiv.org/abs/2401.04088
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — arxiv.org/abs/2101.03961
• Hugging Face Inference Providers — huggingface.co/docs/inference-providers/index
• Unsloth Docs — unsloth.ai/docs
• MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — arxiv.org/abs/2402.14905
• Tiny Aya: Bridging Scale and Multilingual Depth — arxiv.org/abs/2603.11510
• Aya 23: Open Weight Releases to Further Multilingual Progress — arxiv.org/abs/2405.15032
• Self-Instruct: Aligning Language Models with Self-Generated Instructions — arxiv.org/abs/2212.10560
• The Curse of Recursion: Training on Generated Data Makes Models Forget — arxiv.org/abs/2305.17493
• DeepSeek-V3 Technical Report — arxiv.org/abs/2412.19437










