Distillation: Can A Small Model Beat A Bigger Model? @engineerprompt
Distillation: Can A Small Model Beat A Bigger Model?  @engineerprompt
Uploaded October 2026 | Updated November 2026, 3 days ago
Can a 27 billion parameter model teach a 4 billion model to be as good as itself? Last time, RL got Qwen 4B from 52% to 58.4% and then hit a wall. This time I used Qwen 27B as a teacher and tried all three kinds of distillation on two DGX Sparks: training on the teacher's answers, training on its probabilities, and on-policy distillation where the 27B grades every token the 4B writes.

Sponsored by Dex, the talent agent for senior engineers. Free for engineers: meetdex.link/promptengineering-1

Links
- Previous video (RL, the sandbox, the 58% wall): youtu.be/Pl_2TelxjdY
- Code: github.com/PromtEngineer/smoldataenvs-grpo-spark
- SmolDataEnvs (the tasks): huggingface.co/datasets/FineEnvs/SmolDataEnvs
- Thinking Machines, On-Policy Distillation: thinkingmachines.ai/blog/on-policy-distillation
- Qwen3 Technical Report (strong-to-weak distillation): arxiv.org/abs/2505.09388
- Hinton, Vinyals, Dean (2015), Distilling the Knowledge in a Neural Network: arxiv.org/abs/1503.02531
- On-policy distillation papers: MiniLLM arxiv.org/abs/2306.08543 · GKD arxiv.org/abs/2306.13649

Chapters
00:00 - Introduction & The Distillation Idea
02:21 - Benchmark Setup & Small Data Environments
03:35 - Why Reinforcement Learning Hit a Wall
04:40 - The 3 Ways to Distill LLMs
08:04 - Hardware Setup (Dual DGX Spark)
08:37 - Method 1: Training on Teacher Answers
10:00 - Sponsor Break
11:00 - Method 2: Distilling Teacher Probabilities
13:30 - Method 3: Live On-Policy Distillation
16:42 - Comparing All 6 Experiment Results
17:27 - The 58% Ceiling & What the Small Model Actually Knows
19:07 - The 4-Tries Headroom Test
20:24 - Hardware & Practical Implementation Tips
21:21 - Recommendations & Which Method to Use
22:19 - Next Steps: Testing 2B and 9B Models

#distillation #qwen #llm #reinforcementlearning #localai #dgxspark
Distillation: Can A Small Model Beat A Bigger Model?Claude Codes New Agents Team Are Absolutely InsaneStop Building AI Agents the Old WayQwen 27B on 6GB VRAM...Gemini 4 Carbon: The New FrontierGemini 3 Flash is here ...Google Goes All-In on Vibe Coding with AI StudioClaude Can Now Build Its Own Harness... For Every TaskGemini 4 Carbon Is Finally Here, And Its a Beast!Harness² Explained in 60 SecondsThe New Gemini Text to SpeechGPT 5.2 is here.. the biggest update yet?
Prompt Engineering |

Distillation: Can A Small Model Beat A Bigger Model?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER