Training Agents 2: Live tutorial on model distillation for training custom agents. @HuggingFace
Training Agents 2: Live tutorial on model distillation for training custom agents.  @HuggingFace
Uploaded July 2026 | Updated November 2026, 3 days ago
In this live session, we'll cover how to transfer capability from a teacher model to a smaller student through distillation. We'll work through supervised fine-tuning on teacher-generated data, then on-policy and online methods where the teacher scores the student live, then self-distillation where the model teaches itself. Each one runs in TRL.
What we'll cover:

- What distillation is, and the four axes that organize it: signal, data source, timing, and teacher identity
- White-box vs black-box: distilling from open weights vs strings
- Off-policy distillation: generate from the teacher, then SFT on the outputs
- On-policy distillation: sample from the student, score with the teacher in the loop
- Distillation as reinforcement learning: the KD distance as a dense, token-level reward
- Self-distillation: the model as its own teacher, and when that beats a stronger one

01:04 - Introduction to the stream and distillation for training agents.
02:51 - Overview of the session’s running order (off-policy vs. on-policy distillation).
04:06 - Understanding the three main phases of training: pre-training, mid-training, and post-training.
05:22 - Why distillation is useful for smaller, more efficient models.
06:01 - Recap of Supervised Fine-Tuning (SFT) and how it fits into distillation.
08:31 - Breaking down distillation into four axes: Signal, Data, Timing, and Teacher.
09:56 - Hard labels vs. Soft labels (SFT vs. Logit-based distillation).
12:50 - Comparing Reinforcement Learning (RL), off-policy learning, and on-policy learning.
17:16 - Deep dive into off-policy distillation.
20:00 - Implementing off-policy distillation in TRL (using the GKD trainer).
26:40 - Reproducing experiments: training a smaller coding agent via distillation.
32:01 - Deep dive into on-policy distillation and the reverse KL loss.
38:49 - Setting lambda and beta parameters for on-policy distillation in TRL.
48:37 - Introduction to self-distillation (Simple, SDPO, and SDFT).
56:52 - The limits and ceilings of distillation.
58:12 - Final recap and transition to future sessions on Reinforcement Learning.
59:51 - Q&A: Addressing viewer questions on multi-teacher models and closed-model distillation.

Repo: github.com/burtenshaw/training-agents

Slides: docs.google.com/presentation/d/1Vo1AOj1FRemm2NrC9z2RspeJC9VInflQiKeSzr11Kw4/edit?usp=sharing

This is part of the Training Agents series: using coding agents to design, run, monitor, and review post-training experiments, while training models to become better agents.

#TRL #HuggingFace #PostTraining #AIAgents #Distillation
Training Agents 2: Live tutorial on model distillation for training custom agents.Local AI 202: Get Massive Speed-ups with Speculative DecodingBuild Your Own Open-Source Dots with Pi and TelegramOne Minute Gradio #2: Event ChainingHow LLMs Actually Generate TextHugging Face Journal Club: Training AI Scientists to Replicate ResearchHow to contribute to Open Source - 7 EASY steps 🤗Create Your Own Gradio Component - Part 1Evaluate Policy: LeRobot Tutorial #5 by Simon AlibertTransformers.js v4.3: Structured Output in the browserHuggingChat | Chat with Open ModelsI Turned Structured Output Into a Game 🎮
Hugging Face |

Training Agents 2: Live tutorial on model distillation for training custom agents.

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER