Uploaded July 2023 | Updated September 2026, 3 weeks ago
Welch Labs
|
July 10, 2023
🔇 TAP TO UNMUTE
![Learning To See [Part 9: Bias Variance Throwdown]](https://i.ytimg.com/vi/yLwZEuybaqE/mqdefault.jpg "Learning To See [Part 9: Bias Variance Throwdown]
In this series, well explore the complex landscape of machine learning and artificial intelligence through one example from the field of computer vision: using a decision tree to count the number of fingers in an image. Its gonna be crazy.
Supporting Code: https://github.com/stephencwelch/LearningToSee
welchlabs.com
@welchlabs")
![Self-Driving Cars [S1E3: AI Failure?]](https://i.ytimg.com/vi/yaYER2M8dcs/mqdefault.jpg "Self-Driving Cars [S1E3: AI Failure?]
PATREON: https://www.patreon.com/welchlabs
TWITTER: @welchlabs
MORE: http://www.welchlabs.com
CODE: https://github.com/stephencwelch/self_driving_cars
FURTHER READING + REFEERENCES
For a bayesian approach to the same reliability testing math: https://stats.stackexchange.com/questions/73645/how-do-you-derive-the-success-run-theorem-from-the-traditional-form-of-bayes-the
Rand Report: https://www.rand.org/pubs/research_reports/RR1478.html
Zeiler and Fungus 2013: https://arxiv.org/abs/1311.2901
WHO top 10 causes of death: https://www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death
WHO injuries and violence facts: https://apps.who.int/iris/bitstream/handle/10665/44288/9789241599375_eng.pdf%3Bjsessionid=7558B4BDC59A52BB94AEC68A0AD79EE8?sequence=1
Bojarski et al: https://arxiv.org/abs/1704.07911
Dean Pomerleau: http://www.cs.cmu.edu/~bhamner/neuralNetworksProject/pomerleau_dean.pdf
Dean Pomerleau: https://www.ri.cmu.edu/pub_files/pub3/pomerleau_dean_1991_1/pomerleau_dean_1991_1.pdf
Dean Pomerleau: https://www.ri.cmu.edu/publications/life-in-the-fast-lane-the-evolution-of-an-adaptive-vehicle-control-system/
Nvidia End to End: https://arxiv.org/abs/1604.07316
Way ChauffeurNet: https://arxiv.org/abs/1812.03079
VIDEO REFERENCES
Amnon Shashua Talk: https://www.youtube.com/watch?v=GCMXXXmxG-I
NAVLAB: https://www.youtube.com/watch?v=rV0H7u6tmIk
MUSIC
https://www.premiumbeat.com/royalty-free-tracks/jazz-manouche-forever
https://www.premiumbeat.com/royalty-free-tracks/out-of-the-woods
https://www.premiumbeat.com/royalty-free-tracks/wanderer
SPECIAL THANKS TO
Tony Fast
Krish Ravindranath
Karthik Naga
Alex Mones
Michael Grabchak
- AND -
Sid Sarasvati
Ross Hanson
Vin Soma
Antoine Pintout
Jaewon Jung
Raphael J Vasquez
AJ Englehardt
Nate Fuller
Hitoshi Yamauchi")
![Imaginary Numbers Are Real [Part 6: The Complex Plane]](https://i.ytimg.com/vi/z5IG_6_zPDo/mqdefault.jpg "Imaginary Numbers Are Real [Part 6: The Complex Plane]
For full problem statement, check out:
http://www.welchlabs.com/blog/2015/10/2/imaginary-numbers-are-real-part-6-the-complex-plane
More information and resources: http://www.welchlabs.com
Imaginary numbers are not some wild invention, they are the deep and natural result of extending our number system. Imaginary numbers are all about the discovery of numbers existing not in one dimension along the number line, but in full two dimensional space. Accepting this not only gives us more rich and complete mathematics, but also unlocks a ridiculous amount of very real, very tangible problems in science and engineering.
Part 1: Introduction
Part 2: A Little History
Part 3: Cardans Problem
Part 4: Bombellis Solution
Part 5: Numbers are Two Dimensional
Part 6: The Complex Plane
Part 7: Complex Multiplication
Part 8: Math Wizardry
Part 9: Closure
Part 10: Complex Functions
Part 11: Wandering in Four Dimensions
Part 12: Riemanns Solution
Part 13: Riemann Surfaces
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.")
![What the Books Get Wrong about AI [Double Descent]](https://i.ytimg.com/vi/z64a7USuGX0/mqdefault.jpg "What the Books Get Wrong about AI [Double Descent]
Huge thanks to KiwiCo for sponsoring today’s video! Go to https://www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first monthly crate or for 20% off your first Panda Crate!
New Book Available Now! The Welch Labs Illustrated Guide to AI (30:47):
https://www.welchlabs.com/resources/ai-book-ezrzm-msrmc
Sections
0:00 - Intro
3:43 - AlexNet & Overfitting
5:19 - Overfitting
6:45 - Rethinking Generalization
11:05 - KiwiCo is Awesome
12:28 - The Double Descent Hypothesis
13:57 - Double Descent is Real!
16:01 - Double Descent with Polynomial Curvefitting?!
20:36 - But why?
22:35 - Should I throw out my books?
24:28 - The Bias-Variance Tradeoff
28:30 - My take
30:47 - I’ve written a new book on AI!
Books with U-shaped test set error curves:
Murphy, Kevin P. Probabilistic machine learning: an introduction. MIT press, 2022.
Goodfellow, Ian, et al. *Deep learning*. Vol. 1. No. 2. Cambridge: MIT press, 2016.
Russell, Stuart Jonathan, and Peter Norvig, eds. *Prentice Hall series in artificial intelligence*. Englewood Cliffs, NJ:: Prentice Hall, 1995.
Bishop, Christopher M., and Nasser M. Nasrabadi. *Pattern recognition and machine learning*. Vol. 4. No. 4. New York: springer, 2006.
Learning, Machine. Tom mitchell. *Publisher: McGraw Hill* (1997): 31.
Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning. (2009).
Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. An introduction to statistical learning. (2009).
Abu-Mostafa, Yaser S., Malik Magdon-Ismail, and Hsuan-Tien Lin. *Learning from data*. Vol. 4. New York: AMLBook, 2012.
MacKay, David JC. *Information theory, inference and learning algorithms*. Cambridge university press, 2003.
Harvard Team’s code & results:
https://gitlab.com/harvard-machine-learning/double-descent
Great repo showing polynomial double descent:
https://github.com/RylanSchaeffer/Stanford-AI-Alignment-Double-Descent-Tutorial
Technical Notes
- 26:25 For these linear fits, we’re using N=15 instead of N=5 points. This increases the bias and reduces the variance of these fits, making the bias variance trade-off more clear, but also pushes out the interpolation threshold. Full results are here: https://github.com/stephencwelch/manim_videos/blob/master/_2025/generalization/Final Video Polynomial Examples.ipynb
- 27:38 It’s tricky to show the full bias-variance results here, as the variance explodes ad Degree=4. Instead we’ve chosen to show qualitative breakdowns, showing which terms dominate the overall error at each degree. Full results can be seen here: https://github.com/stephencwelch/manim_videos/blob/master/_2025/generalization/Final%20Video%20Polynomial%20Examples.ipynb
Special Thanks to Patrons https://www.patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darrell Thomas, Javier Soto, U007D, Caleb Begly, Rick Rubenstein, Brent Hunsaker, Dan Patterson, Tchsurvives, Alex Adai, Walter Reade, Zyansheep, Walter Reade, Duncan Stannett, Reginald Carey, Jean-Manuel Izaret, dh71633, Adrian Rodriguez, Dimitar Stojanovski, Michael Harder, Peter Maldonado, Emily Pesce, David Johnston, Insang Song, FaeTheWolf, Stephen Taylor, KittenKaboodle, EMatter, PATRICKMCCORMACK, John Beahan, Cameron, Cole Jones, Garrett Thornburg, Jeroen W, Rohit Sharma, GlennB, Emmanuel Cortes, Katie Quinn, Karina C, Cakra WW, Mike Ton, Eric Gometz, MacCallister Higgins, Niko Drossos, David Eraso, Tom Zehle, Steve, Brian Lineburg, rjbl, Michael Loh, Perry Vais, Bengal0, Farhad Manjoo, Sara Chipps, Ellis Driscoll, William Taysom, Will Harmon, CK, Abdullah, Peter Cho, Leo Nikora, Griffin Smith, Ash Katnoria, Alex, Markus Hays Nielsen
Special thanks to: Mikhail Belkin, Preetum Nakkiran, Emily Zhang, Varun Reddy
Code for Welch Labs Videos: https://github.com/stephencwelch/manim_videos
Written by: Stephen Welch
Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu
Premium Beat IDs
EEDYZ3FP44YX8OWT
CFAQJOTYQHT7JYIT")
