Uploaded February 2020 | Updated September 2026, 1 week ago
In this video we dive into the nitty-gritty details of the math behind XGBoost trees. We derive the equations for the Output Values from the leaves as well as the Similarity Score. Then we show how these general equations are customized for Regression or Classification by their respective Loss Functions. If you make it to the end, you will be approximately 22% smarter than you are now! :)
NOTE: This StatQuest assumes that you are already familiar with...
XGBoost Part 1: XGBoost Trees for Regression: youtu.be/OtD8wVaFm6E
XGBoost Part 2: XGBoost Trees for Classification: youtu.be/8b1JEDvenQU
Gradient Boost Part 1: Regression Main Ideas: youtu.be/3CC4N4z3GJc
Gradient Boost Part 2: Regression Details:youtu.be/2xudPOBz-vs
Gradient Boost Part 3: Classification Main Ideas: youtu.be/jxuNLH5dXCs
Gradient Boost Part 4: Classification Details: youtu.be/StWY5QWMXCw
...and Ridge Regression: youtu.be/Q81RR3yKn30
Also note, this StatQuest is based on the following sources:
The original XGBoost manuscript: arxiv.org/pdf/1603.02754.pdf
The original XGBoost presentation: https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf
And the XGBoost Documentation: xgboost.readthedocs.io/en/latest/index.html
Last but not least, I want to extend a special thanks to Giuseppe Fasanella and Samuel Judge for thoughtful discussions and helping me understand the math.
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index
If you'd like to support StatQuest, please consider...
Patreon: patreon.com/statquest
...or...
YouTube Membership: youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join
...buying one of my books, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/statquest-store
...or just donating to StatQuest!
paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
twitter.com/joshuastarmer
Corrections:
1:16 The Lambda should be outside of the square brackets.
#statquest #xgboost
In this video we dive into the nitty-gritty details of the math behind XGBoost trees. We derive the equations for the Output Values from the leaves as well as the Similarity Score. Then we show how these general equations are customized for Regression or Classification by their respective Loss Functions. If you make it to the end, you will be approximately 22% smarter than you are now! :)
NOTE: This StatQuest assumes that you are already familiar with...
XGBoost Part 1: XGBoost Trees for Regression: youtu.be/OtD8wVaFm6E
XGBoost Part 2: XGBoost Trees for Classification: youtu.be/8b1JEDvenQU
Gradient Boost Part 1: Regression Main Ideas: youtu.be/3CC4N4z3GJc
Gradient Boost Part 2: Regression Details:youtu.be/2xudPOBz-vs
Gradient Boost Part 3: Classification Main Ideas: youtu.be/jxuNLH5dXCs
Gradient Boost Part 4: Classification Details: youtu.be/StWY5QWMXCw
...and Ridge Regression: youtu.be/Q81RR3yKn30
Also note, this StatQuest is based on the following sources:
The original XGBoost manuscript: arxiv.org/pdf/1603.02754.pdf
The original XGBoost presentation: https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf
And the XGBoost Documentation: xgboost.readthedocs.io/en/latest/index.html
Last but not least, I want to extend a special thanks to Giuseppe Fasanella and Samuel Judge for thoughtful discussions and helping me understand the math.
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index
If you'd like to support StatQuest, please consider...
Patreon: patreon.com/statquest
...or...
YouTube Membership: youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join
...buying one of my books, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/statquest-store
...or just donating to StatQuest!
paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
twitter.com/joshuastarmer
Corrections:
1:16 The Lambda should be outside of the square brackets.
#statquest #xgboost
