Uploaded June 2023 | Updated September 2026, 1 week ago
New videos coming soon!
TikTok:
tiktok.com/@welchlabs
Patreon:
patreon.com/welchlabs
Welch Labs
|
It's been a while...
🔇 TAP TO UNMUTE
![Neural Networks Demystified [Part 7: Overfitting, Testing, and Regularization]](https://i.ytimg.com/vi/S4ZUwgesjS8/mqdefault.jpg "Neural Networks Demystified [Part 7: Overfitting, Testing, and Regularization]
Weve built and trained our neural network, but before we celebrate, we must be sure that our model is representative of the real world.
Supporting Code:
https://github.com/stephencwelch/Neural-Networks-Demystified
Nate Silvers Book: http://www.amazon.com/Signal-Noise-Many-Predictions-Fail/dp/159420411X/ref=sr_1_1?ie=UTF8&qid=1421442340&sr=8-1&keywords=signal+and+the+noise
Caltech Machine Learning Course: https://work.caltech.edu/telecourse.html
And the lecture shown: http://youtu.be/Dc0sr0kdBVI?t=56m52s
In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday.
Part 1: Data + Architecture
Part 2: Forward Propagation
Part 3: Gradient Descent
Part 4: Backpropagation
Part 5: Numerical Gradient Checking
Part 6: Training
Part 7: Overfitting, Testing, and Regularization
@stephencwelch
welchlabs.com")
![Imaginary Numbers Are Real [Part 1: Introduction]](https://i.ytimg.com/vi/T647CGsuOVU/mqdefault.jpg "Imaginary Numbers Are Real [Part 1: Introduction]
For early access to new videos and other perks:
https://www.patreon.com/welchlabs
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.
Imaginary numbers are not some wild invention, they are the deep and natural result of extending our number system. Imaginary numbers are all about the discovery of numbers existing not in one dimension along the number line, but in full two dimensional space. Accepting this not only gives us more rich and complete mathematics, but also unlocks a ridiculous amount of very real, very tangible problems in science and engineering.
Part 1: Introduction
Part 2: A Little History
Part 3: Cardans Problem
Part 4: Bombellis Solution
Part 5: Numbers are Two Dimensional
Part 6: The Complex Plane
Part 7: Complex Multiplication
Part 8: Math Wizardry
Part 9: Closure
Part 10: Complex Functions
Part 11: Wandering in Four Dimensions
Part 12: Riemanns Solution
Part 13: Riemann Surfaces
Thanks to viewer David O for this correction:
The claim that Euler didn’t know what to do with negative numbers, or thought they were greater than infinity, is a misinterpretation of his On Divergent Series paper. Euler argued that, for infinite series, the word “sum” should mean the original finite expression from which the series originates. For example, applying the geometric series formula 1/(1–r) = 1 + r + r^2 + r^3 + … with r = 2 gives the formal result –1 = 1 + 2 + 4 + 8 + …”. However, he did not mean that negative numbers themselves are infinite, nor that such “sums” are equal in the ordinary arithmetic sense for divergent series.
Source: Paragraphs 1-12 of https://arxiv.org/pdf/1808.02841
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.")
![The Dark Matter of AI [Mechanistic Interpretability]](https://i.ytimg.com/vi/UGO_Ehywuxc/mqdefault.jpg "The Dark Matter of AI [Mechanistic Interpretability]
Take your personal data back with Incogni! Use code WELCHLABS at the link below and get 60% off an annual plan: http://incogni.com/welchlabs
Welch Labs Imaginary Numbers Book!
https://www.welchlabs.com/resources/imaginary-numbers-book
Welch Labs Posters:https://www.welchlabs.com/resources
Special Thanks to Patrons https://www.patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman
My Gemma walkthrough notebook: https://colab.research.google.com/drive/1Y68yNr5TcHr4G5RJ0QHZhKkDe55AUkVj?usp=sharing
Most animations made with Manim: https://github.com/3b1b/manim
References and Further Reading
Chris Olah’s original “Dark Matter of Neural Networks” post: https://transformer-circuits.pub/2024/july-update/index.html#dark-matter
Great recent interview with Chris Olah: https://www.youtube.com/watch?v=ugvHCXCOmm4
Gemma Scope: https://arxiv.org/pdf/2408.05147
Experiment with SAEs yourself here! https://www.neuronpedia.org/
Relevant work from the Anthropic team:
https://transformer-circuits.pub/2022/toy_model/index.html
https://transformer-circuits.pub/2023/monosemantic-features
https://transformer-circuits.pub/2024/scaling-monosemanticity/
Excellent intro Mechanistic Interpretability: https://arena3-chapter1-transformer-interp.streamlit.app/%5B1.2%5D_Intro_to_Mech_Interp
Neel Nanda’s Mechanistic Interpretability Explainer: https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J
Transformer Lens: https://github.com/TransformerLensOrg/TransformerLens
SAE Lens: https://jbloomaus.github.io/SAELens/
Technical Notes
1. There are more advanced and more meaningful ways to map mid layer vectors to outputs, see: https://arxiv.org/pdf/2303.08112, https://neuralblog.github.io/logit-prisms/, https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
2. The 6x2304 matrix is actually 7x2304, we’re ignoring the /bos token.
3. Gemma also includes positional embeddings and lots and lots of normalization layers, which we didn’t really cover
4. I’m conflating tokens and words sometimes, in this example each word is a token, so we don’t have to worry about it too much
5. The “_” characters represent spaces in the token strings
CFAQJOTYQHT7JYIT")
