Uploaded January 2025 | Updated September 2026, 2 weeks ago

---

In this deep dive video, we zoom in on model distillation, an advanced technique to build high-performance small language models at a reasonable cost. First, we explain what a model distillation is. Then, we introduce two popular strategies for distillation, logits distillation and hidden states distillation. We study in detail how they work and how they're implemented in the Arcee DistillKit open-source library. Finally, we look at two Arcee models built with distillation, Arcee SuperNova 70B and Arcee SuperNova Medius 14B.

Note: my calculation at 18:45 is wrong. It's 2.3 Tera tokens, not 2.3 Peta tokens. Sorry about that 🤡

If you’d like to understand how Arcee AI can help your organization build scalable and cost-efficient AI solutions, please get in touch at sales@arcee.ai or by booking a demo at arcee.ai/book-a-demo.

⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. You can also follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️

* Slides: fr.slideshare.net/slideshow/deep-dive-model-distillation-with-distillkit/274619548
* DistillKit: github.com/arcee-ai/DistillKit

00:00 Introduction
00:30 What is model distillation?
04:55 Model distillation with DistillKit
11:20 Logits distillation
20:10 Logits distillation with DistillKit
26:10 Hidden states distillation
31:35 Hidden states distillation with DistillKit
36:00 Pros and cons
40:32 Distillation example: Arcee SuperNova 70B
42:50 Distillation example: Arcee SuperNova Medius 14B
44:40 Conclusion