Uploaded June 2024 | Updated November 2026, 9 hours ago
Direct Preference Optimization (DPO) is a method used for training Large Language Models (LLMs). DPO is a direct way to train the LLM without the need for reinforcement learning, which makes it more effective and more efficient.
Learn about it in this simple video!
This is the third one in a series of 4 videos dedicated to the reinforcement learning methods used for training LLMs.
Full Playlist: youtube.com/playlist?list=PLs8w1Cdi-zvYviYYw_V3qe6SINReGF5M-
Video 0 (Optional): Introduction to deep reinforcement learning youtube.com/watch?v=SgC6AZss478
Video 1: Proximal Policy Optimization youtube.com/watch?v=TjHH_--7l8g
Video 2: Reinforcement Learning with Human Feedback youtube.com/watch?v=Z_JUqJBpVOk
Video 3 (This one!): Deterministic Policy Optimization
00:00 Introduction
01:08 RLHF vs DPO
07:19 The Bradley-Terry Model
11:25 KL Divergence
16:32 The Loss Function
14:36 Conclusion
Get the Grokking Machine Learning book!
manning.com/books/grokking-machine-learning
Discount code (40%): serranoyt
(Use the discount code on checkout)
Direct Preference Optimization (DPO) is a method used for training Large Language Models (LLMs). DPO is a direct way to train the LLM without the need for reinforcement learning, which makes it more effective and more efficient.
Learn about it in this simple video!
This is the third one in a series of 4 videos dedicated to the reinforcement learning methods used for training LLMs.
Full Playlist: youtube.com/playlist?list=PLs8w1Cdi-zvYviYYw_V3qe6SINReGF5M-
Video 0 (Optional): Introduction to deep reinforcement learning youtube.com/watch?v=SgC6AZss478
Video 1: Proximal Policy Optimization youtube.com/watch?v=TjHH_--7l8g
Video 2: Reinforcement Learning with Human Feedback youtube.com/watch?v=Z_JUqJBpVOk
Video 3 (This one!): Deterministic Policy Optimization
00:00 Introduction
01:08 RLHF vs DPO
07:19 The Bradley-Terry Model
11:25 KL Divergence
16:32 The Loss Function
14:36 Conclusion
Get the Grokking Machine Learning book!
manning.com/books/grokking-machine-learning
Discount code (40%): serranoyt
(Use the discount code on checkout)










