Uploaded February 2026 | Updated September 2026, 3 weeks ago
Welch Labs
|
Should KiwiCo do a perceptron kit!?
🔇 TAP TO UNMUTE
![Learning to See [Part 7: There is no f]](https://i.ytimg.com/vi/klWUOO4sHaA/mqdefault.jpg "Learning to See [Part 7: There is no f]
In this series, well explore the complex landscape of machine learning and artificial intelligence through one example from the field of computer vision: using a decision tree to count the number of fingers in an image. Its gonna be crazy.
Supporting Code: https://github.com/stephencwelch/LearningToSee
welchlabs.com
@welchlabs")
![ChatGPT is made from 100 million of these [The Perceptron]](https://i.ytimg.com/vi/l-9ALe3U-Fg/mqdefault.jpg "ChatGPT is made from 100 million of these [The Perceptron]
Go to https://drinkag1.com/welchlabs to subscribe and save $20 off your first subscription of AG1! Thanks to AG1 for sponsoring todays video.
Imaginary Numbers book is back in stock! Update at 23:11
https://www.welchlabs.com/resources/imaginary-numbers-book
Welch Labs Posters:
https://www.welchlabs.com/resources
Perceptrons designs from viewers:
https://sites.google.com/view/jacklangsdorf-funprojects/home/perceptron?authuser=0
https://github.com/rdb64-hobbies/Perceptron/blob/main/BUILDING.md
Really nice Perceptron simulators built by viewers:
https://github.com/srives/Perceptron
https://priyangsubanerjee.github.io/perceptron-simulator/
Special Thanks to Patrons https://www.patreon.com/welchlabs
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin, Nicolas baumann, Jason Singh, Robert Riley, vornska, Barry Silverman
References
Rumelhart, D. E., Mcclelland, J. L. (1987). Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations. United Kingdom: Penguin Random House LLC.
Talking Nets: An Oral History of Neural Networks. (2000). United Kingdom: MIT Press.
Prince, S. J. (2023). Understanding Deep Learning. United Kingdom: MIT Press.
Crevier, D. (1993). AI : the tumultuous history of the search for artificial intelligence. New York: Basic Books.
Cat and dog face dataset: https://www.kaggle.com/datasets/andrewmvd/animal-faces?resource=download
Minsky, M., Papert, S. (2017). Perceptrons: An Introduction to Computational Geometry. United Kingdom: MIT Press.
Widrow, Bernard, and Michael A. Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. *Proceedings of the IEEE* 78.9 (1990): 1415-1442.
Olazaran, Mikel. A sociological history of the neural network controversy. *Advances in computers*. Vol. 37. Elsevier, 1993. 335-425.
Widrow, Bernard. Generalization and information storage in networks of adaline neurons. *Self-organizing systems* (1962): 435-461.
Widrow, Bernard. Thinking about thinking: the discovery of the LMS algorithm. *IEEE Signal Processing Magazine* 22.1 (2005): 100-106.
Technical Notes
Method for counting neurons in ChatGPT: Starting with GPT-2 implementation here: https://github.com/karpathy/build-nanogpt/blob/master/train_gpt2.py - keys, queries, and values are implemented in Linear layers with n_embd inputs and 3*n_embd outputs, where n_embd is the embedding dimension. Output projection layer has n_embd and n_embd outputs. So a single attention layer will have ~4*n_embd neurons. GPT-3 has an embedding dimension of 12,288, so each attention layer has ~49,152 neurons. Each MLP block has n_embd inputs, 4*n_embd hidden units, and n_embd outputs, so ~5*n_embd total neurons, or ~61,440. Total neuron count for GPT-3 is then 96*(49,152+61,440)=10,616,832, ignoring initial embedding and final unembedding. Finally, GPT-4 reportedly has ~1.8 Trillion parameters (https://semianalysis.com/2023/07/10/gpt-4-architecture-infrastructure/), making it ~10x larger than GPT-3. Note that GPT-4 is reportedly a mixture of experts, and not all experts are used for each inference, so it appears that not all 1.8 trillion parameters are used for a given inference call. Assuming that ~10x the parameters means 10x the neurons, then GPT-4 should have ~100M neurons.
CFAQJOTYQHT7JYIT")
![Waffles And Harmonic Motion [Part 1]](https://i.ytimg.com/vi/nKavBl1EWiI/mqdefault.jpg "Waffles And Harmonic Motion [Part 1]
In this two part series, we dig into understanding and quantifying simple harmonic motion (SHM). We try to figure our why oscillating systems that move the way they do, and what ideas from physics govern this motion.
I have unapologetically stolen much of my approach from Richard Feynmans wonderful physics lectures: http://www.feynmanlectures.caltech.edu/I_toc.html. The approach presented here was largely borrowed from Feynman chapters 9 and 21. I encourage anyone interesting in going to deeper to read Feynmans lectures.
Supporting code: http://nbviewer.ipython.org/github/stephencwelch/Acoustics-To-Deep-Learning/blob/master/Waffles%20and%20Harmonic%20Motion%20%5BPart%20I%5D.ipynb
It can be forked on github: https://github.com/stephencwelch/Acoustics-To-Deep-Learning
For more on this, see www.welchlabs.com/blog
@stephencwelch")
![Neural Networks Demystified [Part 5: Numerical Gradient Checking]](https://i.ytimg.com/vi/pHMzNW8Agq4/mqdefault.jpg "Neural Networks Demystified [Part 5: Numerical Gradient Checking]
When building complex systems like neural networks, checking portions of your work can save hours of headache. Here well check our gradient computations.
Supporting code:
https://github.com/stephencwelch/Neural-Networks-Demystified
Link to excellent Stanford tutorial: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
In this series, we will build and train a complete Artificial Neural Network in python. New videos every other friday.
Part 1: Data + Architecture
Part 2: Forward Propagation
Part 3: Gradient Descent
Part 4: Backpropagation
Part 5: Numerical Gradient Checking
Part 6: Training
Part 7: Overfitting, Testing, and Regularization
@stephencwelch")
![Imaginary Numbers Are Real [Part 10: Complex Functions]](https://i.ytimg.com/vi/pNp8Qf20-sA/mqdefault.jpg "Imaginary Numbers Are Real [Part 10: Complex Functions]
Supporting Code: https://github.com/stephencwelch/Imaginary-Numbers-Are-Real
More information and resources: http://www.welchlabs.com
Imaginary numbers are not some wild invention, they are the deep and natural result of extending our number system. Imaginary numbers are all about the discovery of numbers existing not in one dimension along the number line, but in full two dimensional space. Accepting this not only gives us more rich and complete mathematics, but also unlocks a ridiculous amount of very real, very tangible problems in science and engineering.
Part 1: Introduction
Part 2: A Little History
Part 3: Cardans Problem
Part 4: Bombellis Solution
Part 5: Numbers are Two Dimensional
Part 6: The Complex Plane
Part 7: Complex Multiplication
Part 8: Math Wizardry
Part 9: Closure
Part 10: Complex Functions
Part 11: Wandering in Four Dimensions
Part 12: Riemanns Solution
Part 13: Riemann Surfaces
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.")
