Welch Labs | The moment we stopped understanding AI [AlexNet] @WelchLabsVideo | Uploaded 3 months ago | Updated 1 hour ago
Thanks to KiwiCo for sponsoring today's video! Go to kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
Activation Atlas Posters!
welchlabs.com/resources/5gtnaauv6nb9lrhoz9cp604padxp5o
welchlabs.com/resources/activation-atlas-poster-mixed5b-13x19
welchlabs.com/resources/large-activation-atlas-poster-mixed4c-24x36
welchlabs.com/resources/activation-atlas-poster-mixed4c-13x19
Special thanks to the Patrons:
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti
Welch Labs
Ad free videos and exclusive perks: patreon.com/welchlabs
Watch on TikTok: tiktok.com/@welchlabs
Learn More or Contact: welchlabs.com
Instagram: instagram.com/welchlabs
X: twitter.com/welchlabs
References
AlexNet Paper
proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
Original Activation Atlas Article- explore here - Great interactive Atlas! https://distill.pub/2019/activation-atlas/
Carter, et al., "Activation Atlas", Distill, 2019.
Feature Visualization Article: https://distill.pub/2017/feature-visualization/
`Olah, et al., "Feature Visualization", Distill, 2017.`
Great LLM Explainability work: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
Templeton, et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread, 2024.
“Deep Visualization Toolbox" by Jason Yosinski video inspired many visuals:
youtube.com/watch?v=AgkfIQ4IGaM
Great LLM/GPT Intro paper
arxiv.org/pdf/2304.10557
3B1Bs GPT Videos are excellent, as always:
youtube.com/watch?v=eMlx5fFNoYc
youtube.com/watch?v=wjZofJX0v4M
Andrej Kerpathy's walkthrough is amazing:
youtube.com/watch?v=kCc8FmEb1nY
Goodfellow’s Deep Learning Book
deeplearningbook.org
OpenAI’s 10,000 V100 GPU cluster (1+ exaflop) news.microsoft.com/source/features/innovation/openai-azure-supercomputer
GPT-3 size, etc: Language Models are Few-Shot Learners, Brown et al, 2020.
Unique token count for ChatGPT: cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
GPT-4 training size etc, speculative:
patmcguinness.substack.com/p/gpt-4-details-revealed
semianalysis.com/p/gpt-4-architecture-infrastructure
Historical Neural Network Videos
youtube.com/watch?v=FwFduRA_L6Q
youtube.com/watch?v=cNxadbrN_aI
Errata
1:40 should be: "word fragment is appended to the end of the original input". Thanks for Chris A for finding this one.
Thanks to KiwiCo for sponsoring today's video! Go to kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
Activation Atlas Posters!
welchlabs.com/resources/5gtnaauv6nb9lrhoz9cp604padxp5o
welchlabs.com/resources/activation-atlas-poster-mixed5b-13x19
welchlabs.com/resources/large-activation-atlas-poster-mixed4c-24x36
welchlabs.com/resources/activation-atlas-poster-mixed4c-13x19
Special thanks to the Patrons:
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti
Welch Labs
Ad free videos and exclusive perks: patreon.com/welchlabs
Watch on TikTok: tiktok.com/@welchlabs
Learn More or Contact: welchlabs.com
Instagram: instagram.com/welchlabs
X: twitter.com/welchlabs
References
AlexNet Paper
proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
Original Activation Atlas Article- explore here - Great interactive Atlas! https://distill.pub/2019/activation-atlas/
Carter, et al., "Activation Atlas", Distill, 2019.
Feature Visualization Article: https://distill.pub/2017/feature-visualization/
`Olah, et al., "Feature Visualization", Distill, 2017.`
Great LLM Explainability work: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
Templeton, et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread, 2024.
“Deep Visualization Toolbox" by Jason Yosinski video inspired many visuals:
youtube.com/watch?v=AgkfIQ4IGaM
Great LLM/GPT Intro paper
arxiv.org/pdf/2304.10557
3B1Bs GPT Videos are excellent, as always:
youtube.com/watch?v=eMlx5fFNoYc
youtube.com/watch?v=wjZofJX0v4M
Andrej Kerpathy's walkthrough is amazing:
youtube.com/watch?v=kCc8FmEb1nY
Goodfellow’s Deep Learning Book
deeplearningbook.org
OpenAI’s 10,000 V100 GPU cluster (1+ exaflop) news.microsoft.com/source/features/innovation/openai-azure-supercomputer
GPT-3 size, etc: Language Models are Few-Shot Learners, Brown et al, 2020.
Unique token count for ChatGPT: cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
GPT-4 training size etc, speculative:
patmcguinness.substack.com/p/gpt-4-details-revealed
semianalysis.com/p/gpt-4-architecture-infrastructure
Historical Neural Network Videos
youtube.com/watch?v=FwFduRA_L6Q
youtube.com/watch?v=cNxadbrN_aI
Errata
1:40 should be: "word fragment is appended to the end of the original input". Thanks for Chris A for finding this one.
![The moment we stopped understanding AI [AlexNet]](https://i.ytimg.com/vi/UZDiGooFs54/hqdefault.jpg "The moment we stopped understanding AI [AlexNet]
Thanks to KiwiCo for sponsoring todays video! Go to https://www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
Activation Atlas Posters!
https://www.welchlabs.com/resources/5gtnaauv6nb9lrhoz9cp604padxp5o
https://www.welchlabs.com/resources/activation-atlas-poster-mixed5b-13x19
https://www.welchlabs.com/resources/large-activation-atlas-poster-mixed4c-24x36
https://www.welchlabs.com/resources/activation-atlas-poster-mixed4c-13x19
Special thanks to the Patrons:
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti
Welch Labs
Ad free videos and exclusive perks: https://www.patreon.com/welchlabs
Watch on TikTok: https://www.tiktok.com/@welchlabs
Learn More or Contact: https://www.welchlabs.com/
Instagram: https://www.instagram.com/welchlabs
X: https://twitter.com/welchlabs
References
AlexNet Paper
https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
Original Activation Atlas Article- explore here - Great interactive Atlas! https://distill.pub/2019/activation-atlas/
Carter, et al., Activation Atlas, Distill, 2019.
Feature Visualization Article: https://distill.pub/2017/feature-visualization/
`Olah, et al., Feature Visualization, Distill, 2017.`
Great LLM Explainability work: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
Templeton, et al., Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Transformer Circuits Thread, 2024.
“Deep Visualization Toolbox by Jason Yosinski video inspired many visuals:
https://www.youtube.com/watch?v=AgkfIQ4IGaM
Great LLM/GPT Intro paper
https://arxiv.org/pdf/2304.10557
3B1Bs GPT Videos are excellent, as always:
https://www.youtube.com/watch?v=eMlx5fFNoYc
https://www.youtube.com/watch?v=wjZofJX0v4M
Andrej Kerpathys walkthrough is amazing:
https://www.youtube.com/watch?v=kCc8FmEb1nY
Goodfellow’s Deep Learning Book
https://www.deeplearningbook.org/
OpenAI’s 10,000 V100 GPU cluster (1+ exaflop) https://news.microsoft.com/source/features/innovation/openai-azure-supercomputer/
GPT-3 size, etc: Language Models are Few-Shot Learners, Brown et al, 2020.
Unique token count for ChatGPT: https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken
GPT-4 training size etc, speculative:
https://patmcguinness.substack.com/p/gpt-4-details-revealed
https://www.semianalysis.com/p/gpt-4-architecture-infrastructure
Historical Neural Network Videos
https://www.youtube.com/watch?v=FwFduRA_L6Q
https://www.youtube.com/watch?v=cNxadbrN_aI
Errata
1:40 should be: word fragment is appended to the end of the original input. Thanks for Chris A for finding this one.")
![How To Science [Part 4: Science]](https://i.ytimg.com/vi/Y7kCJtWFpUU/hqdefault.jpg "How To Science [Part 4: Science]
PDF: http://www.welchlabs.com/guides
Support Welch Labs: www.patreon.com/welchlabs
Music:
https://www.premiumbeat.com/royalty-free-tracks/jazz-manouche-forever
https://www.premiumbeat.com/royalty-free-tracks/out-of-the-woods
https://www.premiumbeat.com/royalty-free-tracks/truth-to-tell")
![Imaginary Numbers Are Real [Part 7: Complex Multiplication]](https://i.ytimg.com/vi/YHvR8siIiD0/hqdefault.jpg "Imaginary Numbers Are Real [Part 7: Complex Multiplication]
More information and resources: http://www.welchlabs.com
Imaginary numbers are not some wild invention, they are the deep and natural result of extending our number system. Imaginary numbers are all about the discovery of numbers existing not in one dimension along the number line, but in full two dimensional space. Accepting this not only gives us more rich and complete mathematics, but also unlocks a ridiculous amount of very real, very tangible problems in science and engineering.
Part 1: Introduction
Part 2: A Little History
Part 3: Cardans Problem
Part 4: Bombellis Solution
Part 5: Numbers are Two Dimensional
Part 6: The Complex Plane
Part 7: Complex Multiplication
Part 8: Math Wizardry
Part 9: Closure
Part 10: Complex Functions
Part 11: Wandering in Four Dimensions
Part 12: Riemanns Solution
Part 13: Riemann Surfaces
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.
Want to learn more or teach this series? Check out the Imaginary Numbers are Real Workbook: http://www.welchlabs.com/resources.")
![Waffles And Harmonic Motion [Part II]](https://i.ytimg.com/vi/ZzVpT8w8ILU/hqdefault.jpg "Waffles And Harmonic Motion [Part II]
In this two part series, we dig into understanding and quantifying simple harmonic motion (SHM). We try to figure our why systems that oscillate move the way they do, and what ideas from physics govern this motion.
I have unapologetically stolen much of my approach from Richard Feynmans wonderful physics lectures: http://www.feynmanlectures.caltech.edu/I_toc.html. The approach presented here was largely borrowed from Feynman chapters 9 and 21. I encourage anyone interesting in going to deeper to read Feynmans lectures.
Supporting code: https://github.com/stephencwelch/Acoustics-To-Deep-Learning/blob/master/Waffles%20And%20Harmonic%20Motion%20%5BPart%20II%5D.ipynb
For more, see www.welchlabs.com/blog")
![How to Science [Part 3: Experiments]](https://i.ytimg.com/vi/b-CPQdWU-sI/hqdefault.jpg "How to Science [Part 3: Experiments]
PDF: http://www.welchlabs.com/guides/
Support Welch Labs: https://www.patreon.com/welchlabs
Music:
https://premiumbeat.com/royalty_free_music/songs/jazz-manouche-forever
https://premiumbeat.com/royalty_free_music/songs/razer-trap")
![Learning to See [Part 11: Haystacks on Haystacks]](https://i.ytimg.com/vi/biy2yU3Auc4/hqdefault.jpg "Learning to See [Part 11: Haystacks on Haystacks]
In this series, well explore the complex landscape of machine learning and artificial intelligence through one example from the field of computer vision: using a decision tree to count the number of fingers in an image. Its gonna be crazy.
Supporting Code: https://github.com/stephencwelch/LearningToSee
welchlabs.com
@welchlabs")
![Neural Networks Demystified [Part 1: Data and Architecture]](https://i.ytimg.com/vi/bxe2T-V8XRs/hqdefault.jpg "Neural Networks Demystified [Part 1: Data and Architecture]
Neural Networks Demystified
Part 1: Data and Architecture
@stephencwelch
Supporting Code:
https://github.com/stephencwelch/Neural-Networks-Demystified
In this short series, we will build and train a complete Artificial Neural Network in python. New videos every other friday.
Part 1: Data + Architecture
Part 2: Forward Propagation
Part 3: Gradient Descent
Part 4: Backpropagation
Part 5: Numerical Gradient Checking
Part 6: Training
Part 7: Overfitting, Testing, and Regularization")
![Self Driving Cars [S1E1: The ALV]](https://i.ytimg.com/vi/cExJbbwOfcw/hqdefault.jpg "Self Driving Cars [S1E1: The ALV]
PATREON: https://www.patreon.com/welchlabs
TWITTER: @welchlabs
MORE: http://www.welchlabs.com
CODE: https://github.com/stephencwelch/self_driving_cars
FURTHER READING
Original ALV Paper: http://www.cs.ucsb.edu/~mturk/Papers/ALV.pdf
Great Book on SCI: https://www.amazon.com/Strategic-Computing-Machine-Intelligence-1983-1993/dp/0262529262/ref=sr_1_1?ie=UTF8&qid=1547071120&sr=8-1&keywords=strategic+computing
Great book on the history of AI: https://www.amazon.com/Ai-Daniel-Crevier/dp/0465001041/ref=sr_1_1?ie=UTF8&qid=1547071150&sr=8-1&keywords=ai+the+tumultuous+history+of+the+search+for+artificial+intelligence
VIDEO REFERENCES
Elon Musk at TED: https://www.youtube.com/watch?v=NcqI76Z4t1A
Cadillac Super Cruise: https://www.youtube.com/watch?v rxW68ADldI
Tesla Perception: https://www.youtube.com/watch?v=VG68SKoG7vE
Waymo Perception: https://www.youtube.com/watch?v=B8R148hFxPw
Chris Urmson TED talk: https://www.youtube.com/watch?v=tiwVMrTLUWg
MUSIC
https://www.premiumbeat.com/royalty-free-tracks/girl-power
https://www.premiumbeat.com/royalty-free-tracks/out-of-the-woods
https://www.premiumbeat.com/royalty-free-tracks/language-2
SPECIAL THANKS TO
Tony Fast
Krish Ravindranath
Karthik Naga
Charles Young
Chang Lee
Mathew Turk
- And -
Vin Soma
Raphael J Vasquez
Nate Fuller")
![How to Science [Part 1: Music]](https://i.ytimg.com/vi/d3mHfqd0VZY/hqdefault.jpg "How to Science [Part 1: Music]
pdf:
http://www.welchlabs.com/guides
music: https://www.premiumbeat.com/royalty_free_music/songs/storytelling-piano
patreon:
https://www.patreon.com/welchlabs
correction:
length table should be in cm, to mm!")
