Build A Reasoning Model From Scratch 8: Distillation @SebastianRaschka
Build A Reasoning Model From Scratch 8: Distillation  @SebastianRaschka
Uploaded October 2026 | Updated November 2026, 5 hours ago
In this final video, we look into model distillation as a more efficient alternative to reinforcement learning for developing small reasoning models.

- Reasoning playlist: youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs
- Reasoning Book: amzn.to/4aAKiFY
- Reasoning GitHub repo: github.com/rasbt/reasoning-from-scratch

- LLMs from Scratch book: amzn.to/4fqvn0D
- LLMs from Scratch repo: github.com/rasbt/LLMs-from-scratch
- LLMs from Scratch playlist: youtube.com/watch?v=yAcWnfsZhzo&list=PLTKMiZHVd_2IIEsoJrWACkIxLRdfMlw11&index=1

00:00 Introduction and series recap
01:07 Why distill a reasoning model?
02:51 Soft vs. hard distillation
11:06 On-policy vs. off-policy distillation
18:09 Preparing data for reasoning distillation
22:35 Loading the DeepSeek R1 MATH dataset
28:04 Inspecting reasoning traces and final answers
31:27 Tokenization and chat templates
35:58 Pre-tokenizing the training dataset
39:01 Sequence lengths and the 2,048-token limit
42:38 Creating training and validation splits
45:32 Loading the Qwen3-0.6B base model
46:24 Computing answer-token log probabilities
54:10 Cross-entropy loss for distillation
57:25 Training and validation loss helpers
1:02:03 Implementing the distillation training loop
1:07:26 Running the standalone training script
1:09:41 Plotting training and validation losses
1:11:31 Evaluating checkpoints on MATH-500
1:14:09 Comparing teachers and distillation results
1:16:34 Future directions for reasoning models
1:19:41 Bonus exercises and next steps
Build A Reasoning Model From Scratch 8: DistillationL11.7 Weight Initialization in PyTorch   Code ExampleL8.1 Logistic Regression as a Single-Layer Neural NetworkThe Three Elements of PyTorchL6.2 Understanding Automatic Differentiation via Computation GraphsL17.3 The Log-Var TrickL15.0: Introduction to Recurrent Neural Networks   Lecture OverviewL10.2 Data Augmentation in PyTorchL14.3.2.1 ResNet OverviewBuilding LLMs from the Ground Up: A 3-hour Coding WorkshopThe Big LLM Architecture ComparisonInsights from Finetuning LLMs with Low-Rank Adaptation
Sebastian Raschka |

Build A Reasoning Model From Scratch 8: Distillation

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER