![Why Deep Learning Works Unreasonably Well [How Models Learn Part 3] @WelchLabs](https://i.ytimg.com/vi/qx7hirqgfuU/maxresdefault.jpg)
Uploaded August 2025 | Updated September 2026, 2 weeks ago
Take your personal data back with Incogni! Use code WELCHLABS and get 60% off an annual plan: incogni.com/welchlabs
Welch Labs Guide to AI: welchlabs.com/resources/ai-book-ezrzm-msrmc
New Patreon Rewards 33:31- own a piece of Welch Labs history!
patreon.com/welchlabs
Books & Posters
welchlabs.com/resources
Sections
0:00 - Intro
4:49 - How Incogni Saves Me Time
6:32 - Part 2 Recap
8:10 - Moving to Two Layers
9:15 - How Activation Functions Fold Space
11:45 - Numerical Walkthrough
13:42 - Universal Approximation Theorem
15:45 - The Geometry of Backpropagation
19:52 - The Geometry of Depth
24:27 - Exponentially Better?
30:23 - Neural Networks Demystifed
31:50 - The Time I Quit YouTube
33:31 - New Patreon Rewards!
Special Thanks to Patrons patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darrell Thomas, Javier Soto
References
Simon Prince, Understanding Deep Learning. udlbook.github.io/udlbook
Liang, Shiyu, and Rayadurgam Srikant. "Why deep neural networks for function approximation?." arXiv preprint arXiv:1610.04161 (2016).
Hanin, Boris, and David Rolnick. "Deep relu networks have surprisingly few activation patterns." *Advances in neural information processing systems* 32 (2019).
Hanin, Boris, and David Rolnick. "Complexity of linear regions in deep networks." *International Conference on Machine Learning*. PMLR, 2019.
Fan, Feng-Lei, et al. "Deep relu networks have surprisingly simple polytopes." *arXiv preprint arXiv:2305.09145* (2023).
All Code:
github.com/stephencwelch/manim_videos
100k neuron wide example training code: github.com/stephencwelch/manim_videos/blob/master/_2025/backprop_3/notebooks/Wide%20Training%20Example.ipynb
Code from viewer Hugo Brouwer that achieves 99.5%+ with less than 100 neurons using Fourier Features!
github.com/AgntBrwr/baarle-hertog-fourier-features
Code from viewer Nico Waser that uses 100 neurons:
github.com/Waser2004/Illustrated_-_Guide_to_AI/tree/main/Chapter%204%20-%20Deep%20Learning
Written by: Stephen Welch
Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu
Premium Beat IDs
EEDYZ3FP44YX8OWTe
MWROXNAY0SPXCMBS
CFAQJOTYQHT7JYIT
Take your personal data back with Incogni! Use code WELCHLABS and get 60% off an annual plan: incogni.com/welchlabs
Welch Labs Guide to AI: welchlabs.com/resources/ai-book-ezrzm-msrmc
New Patreon Rewards 33:31- own a piece of Welch Labs history!
patreon.com/welchlabs
Books & Posters
welchlabs.com/resources
Sections
0:00 - Intro
4:49 - How Incogni Saves Me Time
6:32 - Part 2 Recap
8:10 - Moving to Two Layers
9:15 - How Activation Functions Fold Space
11:45 - Numerical Walkthrough
13:42 - Universal Approximation Theorem
15:45 - The Geometry of Backpropagation
19:52 - The Geometry of Depth
24:27 - Exponentially Better?
30:23 - Neural Networks Demystifed
31:50 - The Time I Quit YouTube
33:31 - New Patreon Rewards!
Special Thanks to Patrons patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darrell Thomas, Javier Soto
References
Simon Prince, Understanding Deep Learning. udlbook.github.io/udlbook
Liang, Shiyu, and Rayadurgam Srikant. "Why deep neural networks for function approximation?." arXiv preprint arXiv:1610.04161 (2016).
Hanin, Boris, and David Rolnick. "Deep relu networks have surprisingly few activation patterns." *Advances in neural information processing systems* 32 (2019).
Hanin, Boris, and David Rolnick. "Complexity of linear regions in deep networks." *International Conference on Machine Learning*. PMLR, 2019.
Fan, Feng-Lei, et al. "Deep relu networks have surprisingly simple polytopes." *arXiv preprint arXiv:2305.09145* (2023).
All Code:
github.com/stephencwelch/manim_videos
100k neuron wide example training code: github.com/stephencwelch/manim_videos/blob/master/_2025/backprop_3/notebooks/Wide%20Training%20Example.ipynb
Code from viewer Hugo Brouwer that achieves 99.5%+ with less than 100 neurons using Fourier Features!
github.com/AgntBrwr/baarle-hertog-fourier-features
Code from viewer Nico Waser that uses 100 neurons:
github.com/Waser2004/Illustrated_-_Guide_to_AI/tree/main/Chapter%204%20-%20Deep%20Learning
Written by: Stephen Welch
Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu
Premium Beat IDs
EEDYZ3FP44YX8OWTe
MWROXNAY0SPXCMBS
CFAQJOTYQHT7JYIT
![Why Deep Learning Works Unreasonably Well [How Models Learn Part 3]](https://i.ytimg.com/vi/qx7hirqgfuU/mqdefault.jpg "Why Deep Learning Works Unreasonably Well [How Models Learn Part 3]
Take your personal data back with Incogni! Use code WELCHLABS and get 60% off an annual plan: http://incogni.com/welchlabs
Welch Labs Guide to AI: https://www.welchlabs.com/resources/ai-book-ezrzm-msrmc
New Patreon Rewards 33:31- own a piece of Welch Labs history!
https://www.patreon.com/welchlabs
Books & Posters
https://www.welchlabs.com/resources
Sections
0:00 - Intro
4:49 - How Incogni Saves Me Time
6:32 - Part 2 Recap
8:10 - Moving to Two Layers
9:15 - How Activation Functions Fold Space
11:45 - Numerical Walkthrough
13:42 - Universal Approximation Theorem
15:45 - The Geometry of Backpropagation
19:52 - The Geometry of Depth
24:27 - Exponentially Better?
30:23 - Neural Networks Demystifed
31:50 - The Time I Quit YouTube
33:31 - New Patreon Rewards!
Special Thanks to Patrons https://www.patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darrell Thomas, Javier Soto
References
Simon Prince, Understanding Deep Learning. https://udlbook.github.io/udlbook/
Liang, Shiyu, and Rayadurgam Srikant. Why deep neural networks for function approximation?. arXiv preprint arXiv:1610.04161 (2016).
Hanin, Boris, and David Rolnick. Deep relu networks have surprisingly few activation patterns. *Advances in neural information processing systems* 32 (2019).
Hanin, Boris, and David Rolnick. Complexity of linear regions in deep networks. *International Conference on Machine Learning*. PMLR, 2019.
Fan, Feng-Lei, et al. Deep relu networks have surprisingly simple polytopes. *arXiv preprint arXiv:2305.09145* (2023).
All Code:
https://github.com/stephencwelch/manim_videos
100k neuron wide example training code: https://github.com/stephencwelch/manim_videos/blob/master/_2025/backprop_3/notebooks/Wide%20Training%20Example.ipynb
Code from viewer Hugo Brouwer that achieves 99.5%+ with less than 100 neurons using Fourier Features!
https://github.com/AgntBrwr/baarle-hertog-fourier-features
Code from viewer Nico Waser that uses 100 neurons:
https://github.com/Waser2004/Illustrated Guide_to_AI/tree/main/Chapter%204%20-%20Deep%20Learning
Written by: Stephen Welch
Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu
Premium Beat IDs
EEDYZ3FP44YX8OWTe
MWROXNAY0SPXCMBS
CFAQJOTYQHT7JYIT")
![How to Science [Part 5: Mathematics]](https://i.ytimg.com/vi/sTSNTWKGkbw/mqdefault.jpg "How to Science [Part 5: Mathematics]
pdf: http://www.welchlabs.com/guides
support welch labs:
https://www.patreon.com/welchlabs
music:
https://www.premiumbeat.com/royalty-free-tracks/jazz-manouche-forever
https://www.premiumbeat.com/royalty-free-tracks/out-of-the-wood
shttps://www.premiumbeat.com/royalty-free-tracks/among-the-hills")
![Learning to See [Part 4: Machine Learning]](https://i.ytimg.com/vi/sarVw-iVWgc/mqdefault.jpg "Learning to See [Part 4: Machine Learning]
In this series, well explore the complex landscape of machine learning and artificial intelligence through one example from the field of computer vision: using a decision tree to count the number of fingers in an image. Its gonna be crazy.
Supporting Code: https://github.com/stephencwelch/LearningToSee
welchlabs.com
@welchlabs")
![Learning To See [Part 14: Better Heuristics]](https://i.ytimg.com/vi/tPHImr2sFBM/mqdefault.jpg "Learning To See [Part 14: Better Heuristics]
In this series, well explore the complex landscape of machine learning and artificial intelligence through one example from the field of computer vision: using a decision tree to count the number of fingers in an image. Its gonna be crazy.
Supporting Code: https://github.com/stephencwelch/LearningToSee
welchlabs.com
@welchlabs")
![Yann LeCuns $1B Bet Against LLMs [Part 2]](https://i.ytimg.com/vi/v_jDvpEGTIg/mqdefault.jpg "Yann LeCuns $1B Bet Against LLMs [Part 2]
Huge thanks to KiwiCo for sponsoring today’s video! Go to https://www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first monthly crate or for 20% off your first Panda Crate!
Welch Labs Book: https://www.welchlabs.com/resources/ai-book-ezrzm-msrmc
JEPA Poster: https://www.welchlabs.com/store/subliminal-learning-poster-17x22-xkfa2-7m2hh
Patreon: https://www.patreon.com/c/welchlabs
Part 1: https://www.youtube.com/watch?v=kYkIdXwW2AE
Sections
0:00 - Intro
2:30 - V-JEPA
8:18 - VL-JEPA
14:05 - But what about VLA?
15:13 - My kids love KiwiCo
17:14 - LeCun’s critique of VLA
22:42 - LeWorldModel
30:41 - Hierarchical JEPA
34:55 - My Take
35:57 - The Future of JEPA
38:57 - JEPA Poster & Patreon Update
Special thanks to: Yann LeCun, Chen Delong, Lucas Maes, Mustafa Shukor, Wancong Zhang, Alison Young, and Theo Moutakanni
Big thank you for Yann’s collaborator Randall Balestriero for discussing JEPA at length with me, Randall’s his recommended reading:
- LeJEPA for more theory on why isotropic Gaussian and details on SIGReg: https://arxiv.org/abs/2511.08544
- connection between SSL and spectral embedding (very useful to understand the geometry learned by those models): https://arxiv.org/abs/2205.11508
- LeWorldModel (world model version of LeJEPA): https://arxiv.org/abs/2603.19312
- identifiability theory of LeWorldModel (fresh out of the oven): https://arxiv.org/abs/2605.26379
- and perhaps most important of all, the reason why one should use JEPA and not reconstruction based learning: https://arxiv.org/abs/2402.11337 and https://arxiv.org/abs/2505.12477
Thanks to Ayzaan Wahid for permission to use clips from: https://www.youtube.com/watch?v=PHXQFE-Rteo
Aloha Controller video: https://www.youtube.com/watch?v=VUxFhtGWD7w Under proper reference to “by Zipeng Fu at Stanford University”, permission given.
Pi07: https://www.pi.website/blog/pi07
V-JEPA 2: https://arxiv.org/pdf/2506.09985
VL-JEPA: https://arxiv.org/pdf/2512.10942
LeWorldModel: https://arxiv.org/pdf/2603.19312
Hierarchical World Models: https://arxiv.org/pdf/2604.03208
Image Credits: https://commons.wikimedia.org/wiki/File:Omphalotus_olearius_Mallorca.jpg
Stock videos & Images
Adobe Stock: 256573141, 307085467, 240682884
Technical note about VL-JEPA embeddings. The VL-JEPA team used a SONAR decoder/encoder, which works at the sentence level instead of the token level for the controlled VLM/JEPA comparisons.
PATRONS
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman, Jake Ehrlich, Mitch Jacobs, Lauren Steely, Jeff Eastman, Rodolfo Ibarra, Clark Barrus, Rob Napier, Andrew White, Richard B Johnston, abhiteja mandava, Burt Humburg, Kevin Mitchell, Daniel Sanchez, Ferdie Wang, Tripp Hill, Richard Harbaugh Jr, Prasad Raje, Kalle Aaltonen, Midori Switch Hound, Zach Wilson, Chris Seltzer, Ven Popov, Hunter Nelson, Amit Bueno, Scott Olsen, Johan Rimez, Shehryar Saroya, Tyler Christensen, Beckett Madden-Woods, Darrell Thomas, Javier Soto, U007D, Caleb Begly, Rick Rubenstein, Brent Hunsaker, Dan Patterson, Tchsurvives, Alex Adai, Walter Reade, Zyansheep, Walter Reade, Duncan Stannett, Reginald Carey, Jean-Manuel Izaret, dh71633, Adrian Rodriguez, Dimitar Stojanovski, Michael Harder, Peter Maldonado, Emily Pesce, David Johnston, Insang Song, FaeTheWolf, Stephen Taylor, KittenKaboodle, EMatter, PATRICKMCCORMACK, John Beahan, Cameron, Cole Jones, Garrett Thornburg, Jeroen W, Rohit Sharma, GlennB, Emmanuel Cortes, Katie Quinn, Karina C, Cakra WW, Mike Ton, Eric Gometz, MacCallister Higgins, Niko Drossos, David Eraso, Tom Zehle, Steve, Brian Lineburg, rjbl, Michael Loh, Perry Vais, Bengal0, Farhad Manjoo, Sara Chipps, Ellis Driscoll, William Taysom, Will Harmon, CK, Abdullah, Peter Cho, Leo Nikora, Griffin Smith, Ash Katnoria, Alex, Markus Hays Nielsen, Catherine H., Vi, David Dobáš, Peter Wang, Sina Sohangir, Danny Thomas, Julian Francis, Hans Adler, Jiayu Peng, Weston M, Youssouf da Silva, John Thomas, Samuel Costello, Sam Adams, Bryan Liles, Malaya Zemlya, Karl, Vahe Andonians, Mike Doughty, Larry Novelo, Jonas Acres, Ludicrum Rex, Robert Blumofe, Anthony Z, Alex Zhao, Dan Babitch, Nikko Patten, Sam Adams, Rahul Ravu, Marco Salvi, Ralph Dratman, Brendan Ardagh, Mindaugas Kazlauskas
Supporting code: https://github.com/WelchLabs/videos
Created by: Matthew Cohen, Sam Baskin, Pranav Gundu, Varun Reddy, and Stephen Welch
Content ID: CFAQJOTYQHT7JYIT")
