Uploaded May 2025 | Updated November 2026, 1 day ago
GRPO is what DeepSeek used to train its amazing reasoning model. The biggest innovation comes from using reinforcement learning to get the model to improve itself, as opposed to self supervised learning. Learn all about it in this friendly video!
Other videos in RL for LLms:
Deep Reinforcement Learning: youtube.com/watch?v=SgC6AZss478
Reinforcement Learning with Human Feedback (RLHF): youtube.com/watch?v=Z_JUqJBpVOk
Proximal Policy Optimization (PPO): youtube.com/watch?v=TjHH_--7l8g
Direct Preference Optimization (DPO): youtube.com/watch?v=k2pD3k1485A
Group Relative Policy Optimization (GRPO): youtu.be/XeUB4h1OO1g
Get the Grokking Machine Learning book!
manning.com/books/grokking-machine-learning
Discount code (40%): serranoyt
(Use the discount code on checkout)
00:00 Introduction
00:26 Answering with context
01:40 DeepSeek vs ChatGPT
05:30 The GRPO score
07:05 Averaging over answers and steps
07:38 Quality (Advantage)
10:30 Probability of responses
15:36 Clipping the response
18:21 Not changing the model too much
GRPO is what DeepSeek used to train its amazing reasoning model. The biggest innovation comes from using reinforcement learning to get the model to improve itself, as opposed to self supervised learning. Learn all about it in this friendly video!
Other videos in RL for LLms:
Deep Reinforcement Learning: youtube.com/watch?v=SgC6AZss478
Reinforcement Learning with Human Feedback (RLHF): youtube.com/watch?v=Z_JUqJBpVOk
Proximal Policy Optimization (PPO): youtube.com/watch?v=TjHH_--7l8g
Direct Preference Optimization (DPO): youtube.com/watch?v=k2pD3k1485A
Group Relative Policy Optimization (GRPO): youtu.be/XeUB4h1OO1g
Get the Grokking Machine Learning book!
manning.com/books/grokking-machine-learning
Discount code (40%): serranoyt
(Use the discount code on checkout)
00:00 Introduction
00:26 Answering with context
01:40 DeepSeek vs ChatGPT
05:30 The GRPO score
07:05 Averaging over answers and steps
07:38 Quality (Advantage)
10:30 Probability of responses
15:36 Clipping the response
18:21 Not changing the model too much


![Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction
Correction: At 30:42 I write X = Y. Theyre not equal, what I meant to say is X and Y are identically distributed.
The variance is a measure of how spread out a distribution is. In order to estimate the variance, one takes a sample of n points from the distribution, and calculate the average square deviation from the mean.
However, this doesnt give a good estimate of the variance of the distribution. The best estimate, however, is obtained when dividing by n-1 instead of n.
WHY!?!?!?!?!?!?!?
In this video, we dig deeper into why the variance calculation should be divided by n-1 instead of by n. For this, we use an alternate definition of the variance, which doesnt use the mean in its calculation.
*[0:00] Introduction and Bessels Correction*
- Introducing Bessels Correction and why we divide by ( n-1 ) instead of ( n ) to estimate variance.
*[0:12] Introduction to Variance Calculation*
- Explaining the premise of calculating variance and introducing the concept of estimating variance using a sample instead of the entire population.
*[1:01] Definition of Variance*
- Defining variance as a measure of how much values deviate from the mean and outlining the basic steps of variance calculation.
*[1:52] Introduction to Bessels Correction*
- Discussing why we divide by ( n-1 ) when calculating variance and introducing Bessels Correction.
*[2:35] Challenges of Bessels Correction*
- Sharing personal challenges in understanding the rationale behind Bessels Correction and discussing my research process on the topic.
*[3:20] Alternative Definition of Variance*
- Presenting an alternative definition of variance to aid in understanding Bessels Correction and expressing curiosity about its presence in the literature.
*[4:45] Quick Recap of Mean and Variance*
- Briefly revisiting the concepts of mean and variance, demonstrating how they are calculated with examples, and explaining how variance reflects different distributions.
*[7:05] Sample Mean and Variance Estimation*
- Explaining the challenges of estimating the mean and variance of a distribution using a sample and discussing why sample variance is not a good estimate.
*[8:49] Bessels Correction and Why ( n-1 ) is Used*
- Explaining how Bessels Correction provides a better estimate of variance and why we divide by ( n-1 ) instead of ( n ). Emphasizing the importance of making a correct variance estimate.
*[10:51] Why Better Estimation Matters?*
- Discussing why the original estimate is poor and why making a better estimate is crucial. Explaining the significance of sample mean as a good estimate.
*[13:02] Issues with Variance Estimation*
- Illustrating the problems with variance estimation and demonstrating with examples why using the correct mean is essential for accurate estimates. Explaining the accuracy of estimates made using ( n-1 ).
*[15:04] Introduction to Correcting the Estimate*
- Discussing the underestimated variance and the need for correction in estimation.
*[15:57] Adjusting the Variance Formula*
- Explaining the adjustment in the variance formula by changing the denominator from ( n ) to ( n - 1 ).
*[16:22] Calculation Illustration*
- Demonstrating the calculation process of variance with the adjusted formula using examples.
*[16:57] Better Estimate with Bessels Correction*
- Discussing how the corrected estimate provides a more accurate variance estimation.
*[18:24] New Method for Variance Calculation*
- Introducing a new method for calculating variance without explicitly calculating the mean.
*[20:06] Understanding the Relation between Variance and Variance*
- Explaining the relationship between variance and variance, and how they are related mathematically.
*[21:52] Demonstrating a Bad Calculation*
- Illustrating a flawed method for calculating variance and explaining the need for correction.
*[23:37] The Role of Bessels Correction*
- Explaining why removing unnecessary zeros in variance calculation leads to better estimates, equivalent to Bessels Correction.
*[25:08] Summary of Estimation Methods*
- Summarizing the difference between the flawed and corrected estimation methods for variance.
*[26:02] Importance of Bessels Correction*
- Emphasizing the significance of Bessels Correction for accurate variance estimation, especially with smaller sample sizes.
*[30:19] Mathematical Proof of Variance Relationship*
- Providing two proofs of the relationship between variance and variance, highlighting their equivalence.
*[35:24] Acknowledgments and Conclusion*
Thanks @mkan543 for the summary! Why do we divide by n-1 to estimate the variance? A visual tour through Bessel correction](https://i.ytimg.com/vi/_sdNwY0KKUA/mqdefault.jpg)







