Uploaded July 2026 | Updated November 2026, 26 minutes ago
Training Agents, Session 3: from a fine-tuned baseline to reinforcement learning with GRPO.
In this live session, we'll move past imitation and start training on outcomes. We'll take the model from the earlier sessions and run Group Relative Policy Optimization: sample multiple completions per prompt, score them with a reward function, and use the group's relative scores as the training signal — no value model, no reward model, no critic to train. Everything runs in TRL.
What we'll cover:
- Why GRPO after SFT, and what RL adds that imitation can't
- How GRPO works: group sampling, relative advantages, and why that replaces a critic
- Writing verifiable reward functions for agent tasks
- Running TRL GRPO on Hugging Face Jobs
- Reading the training curves: reward, KL, and completion length
- Common failure modes: reward hacking, collapse, and length gaming
Repo: github.com/burtenshaw/training-agents
Slides: docs.google.com/presentation/d/19v5_HR5B-1CPHuoZ-RXjhgBrGFXGnBNd6TfLhsXWT1c/edit?usp=sharing
00:00 - Introduction and series recap (SFT and Distillation)
01:38 - Introduction to GRPO (Group Relative Policy Optimization)
02:57 - Understanding the learning landscape and weight updates
06:36 - Deep dive into GRPO: Loop, baseline, and guardrails
09:00 - The reinforcement learning loop for LLMs
10:12 - Comparing RL, Off-policy, and On-policy Distillation
15:05 - Anatomy of a single GRPO step
17:15 - Calculating the group baseline and advantage
18:58 - Guardrails: KL Divergence and Clipping
20:51 - Code implementation of rewards and advantages
22:33 - Designing reward functions as contracts
28:22 - The importance of variation in training groups
32:15 - Strategies to prevent reward hacking
35:05 - Comparing GRPO with DPO and PPO
38:36 - GRPO parameters in TRL (Transformer Reinforcement Learning)
40:49 - How to interpret training curves (Reward, Entropy, KL)
44:36 - Experiment 1: Minimal GRPO training with dummy rewards
52:01 - Experiment 2: Training a coding agent with unit tests
1:01:36 - Experiment 3: Analyzing a gameable reward (Reward Hacking)
1:05:12 - Q&A and final recaps
This is part of the Training Agents series: using coding agents to design, run, monitor, and review post-training experiments, while training models to become better agents.
Training Agents, Session 3: from a fine-tuned baseline to reinforcement learning with GRPO.
In this live session, we'll move past imitation and start training on outcomes. We'll take the model from the earlier sessions and run Group Relative Policy Optimization: sample multiple completions per prompt, score them with a reward function, and use the group's relative scores as the training signal — no value model, no reward model, no critic to train. Everything runs in TRL.
What we'll cover:
- Why GRPO after SFT, and what RL adds that imitation can't
- How GRPO works: group sampling, relative advantages, and why that replaces a critic
- Writing verifiable reward functions for agent tasks
- Running TRL GRPO on Hugging Face Jobs
- Reading the training curves: reward, KL, and completion length
- Common failure modes: reward hacking, collapse, and length gaming
Repo: github.com/burtenshaw/training-agents
Slides: docs.google.com/presentation/d/19v5_HR5B-1CPHuoZ-RXjhgBrGFXGnBNd6TfLhsXWT1c/edit?usp=sharing
00:00 - Introduction and series recap (SFT and Distillation)
01:38 - Introduction to GRPO (Group Relative Policy Optimization)
02:57 - Understanding the learning landscape and weight updates
06:36 - Deep dive into GRPO: Loop, baseline, and guardrails
09:00 - The reinforcement learning loop for LLMs
10:12 - Comparing RL, Off-policy, and On-policy Distillation
15:05 - Anatomy of a single GRPO step
17:15 - Calculating the group baseline and advantage
18:58 - Guardrails: KL Divergence and Clipping
20:51 - Code implementation of rewards and advantages
22:33 - Designing reward functions as contracts
28:22 - The importance of variation in training groups
32:15 - Strategies to prevent reward hacking
35:05 - Comparing GRPO with DPO and PPO
38:36 - GRPO parameters in TRL (Transformer Reinforcement Learning)
40:49 - How to interpret training curves (Reward, Entropy, KL)
44:36 - Experiment 1: Minimal GRPO training with dummy rewards
52:01 - Experiment 2: Training a coding agent with unit tests
1:01:36 - Experiment 3: Analyzing a gameable reward (Reward Hacking)
1:05:12 - Q&A and final recaps
This is part of the Training Agents series: using coding agents to design, run, monitor, and review post-training experiments, while training models to become better agents.