Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning @SerranoAcademy
Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning  @SerranoAcademy
Uploaded June 2024 | Updated November 2026, 9 hours ago
Direct Preference Optimization (DPO) is a method used for training Large Language Models (LLMs). DPO is a direct way to train the LLM without the need for reinforcement learning, which makes it more effective and more efficient.
Learn about it in this simple video!

This is the third one in a series of 4 videos dedicated to the reinforcement learning methods used for training LLMs.

Full Playlist: youtube.com/playlist?list=PLs8w1Cdi-zvYviYYw_V3qe6SINReGF5M-

Video 0 (Optional): Introduction to deep reinforcement learning youtube.com/watch?v=SgC6AZss478
Video 1: Proximal Policy Optimization youtube.com/watch?v=TjHH_--7l8g
Video 2: Reinforcement Learning with Human Feedback youtube.com/watch?v=Z_JUqJBpVOk
Video 3 (This one!): Deterministic Policy Optimization

00:00 Introduction
01:08 RLHF vs DPO
07:19 The Bradley-Terry Model
11:25 KL Divergence
16:32 The Loss Function
14:36 Conclusion

Get the Grokking Machine Learning book!
manning.com/books/grokking-machine-learning
Discount code (40%): serranoyt
(Use the discount code on checkout)
Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learningA friendly introduction to Bayes Theorem and Hidden Markov ModelsKolmogorov-Arnold Networks (KANs) - What are they and how do they work?Mean, Variance, Skewness, and Kurtosis - Math for ML with Deeplearning.aiWhy is ChatGPT so bad at telling jokes (yet so good at writing poems?)The Kolmogorov-Arnold TheoremThompson sampling, one armed bandits, and the Beta distributionHow do you minimize a function when you cant take derivatives? CMA-ES and PSOEulers number, derivatives, and the bank at the end of the universeDid AI Solve Navier-Stokes? The $1M Millennium Problem, ExplainedGaussian Mixture ModelsWhat are Transformer Models and how do they work?
Luis Serrano Academy |

Direct Preference Optimization (DPO) - How to fine-tune LLMs directly without reinforcement learning

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER