Gradient descent, how neural networks learn | Chapter 2, Deep learning @3blue1brown

3Blue1Brown | Gradient descent, how neural networks learn | Chapter 2, Deep learning @3blue1brown | Uploaded 6 years ago | Updated 17 minutes ago
Enjoy these videos? Consider sharing one or two.
Help fund future projects: patreon.com/3blue1brown
Special thanks to these supporters: http://3b1b.co/nn2-thanks
Written/interactive form of this series: 3blue1brown.com/topics/neural-networks

This video was supported by Amplify Partners.
For any early-stage ML startup founders, Amplify Partners would love to hear from you via 3blue1brown@amplifypartners.com

To learn more, I highly recommend the book by Michael Nielsen
http://neuralnetworksanddeeplearning.com
The book walks through the code behind the example in these videos, which you can find here:
github.com/mnielsen/neural-networks-and-deep-learning

MNIST database:
http://yann.lecun.com/exdb/mnist

Also check out Chris Olah's blog:
http://colah.github.io
His post on Neural networks and topology is particular beautiful, but honestly all of the stuff there is great.

And if you like that, you'll *love* the publications at distill:
https://distill.pub/

For more videos, Welch Labs also has some great series on machine learning:
youtu.be/i8D90DkCLhI
youtu.be/bxe2T-V8XRs

"But I've already voraciously consumed Nielsen's, Olah's and Welch's works", I hear you say. Well well, look at you then. That being the case, I might recommend that you continue on with the book "Deep Learning" by Goodfellow, Bengio, and Courville.

Thanks to Lisha Li (@lishali88) for her contributions at the end, and for letting me pick her brain so much about the material. Here are the articles she referenced at the end:
arxiv.org/abs/1611.03530
arxiv.org/abs/1706.05394
arxiv.org/abs/1412.0233

Music by Vincent Rubinetti:
vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown

Thanks to these viewers for their contributions to translations
Hebrew: Omer Tuchfeld
Italian: @teobucci

-------------------
Video timeline
0:00 - Introduction
0:30 - Recap
1:49 - Using training data
3:01 - Cost functions
6:55 - Gradient descent
11:18 - More on gradient vectors
12:19 - Gradient descent recap
13:01 - Analyzing the network
16:37 - Learning more
17:38 - Lisha Li interview
19:58 - Closing thoughts
------------------

3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that).

If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended

Various social media stuffs:
Website: 3blue1brown.com
Twitter: twitter.com/3Blue1Brown
Patreon: patreon.com/3blue1brown
Facebook: facebook.com/3blue1brown
Reddit: reddit.com/r/3Blue1Brown

Convolutions | Why X+Y in probability is a beautiful mess

What is backpropagation really doing? | Chapter 3, Deep learning

The determinant | Chapter 6, Essence of linear algebra

Positioned as the hardest question on a Putnam exam (#6, 1992)

$But why would light slow down? | Optics puzzles 3 How the index of refraction arises, and why it depends on color. Quotebook Notebooks: https://3b1b.co/store These lessons are primarily funded directly by viewers: https://3b1b.co/support An equally valuable form of support is to simply share the videos. Looking Glass Universe videos on the index of refraction: https://youtu.be/uo3ds0FVpXs Much of this video is based on the following Feynmann lecture https://www.feynmanlectures.caltech.edu/I_31.html The explanation for why the phase of a wave produced by a plane of oscillating charges is a quarter phase behind the wave of a charge in the center of that plane, and hence a quarter phase behind that of a light wave inducing the oscillations, is given in the previous chapter: https://www.feynmanlectures.caltech.edu/I_30.html Sections: 0:00 - The standard explanation 3:14 - The plan 5:09 - Phase kicks 8:25 - What causes light? 13:20 - Adding waves 16:40 - Modeling the charge oscillation 20:59 - The driven harmonic oscillator 26:57 - End notes Thanks to these viewers for their contributions to translations German: Ole Hebrew: Alon Altman, Omer Tuchfeld Hindi: Saurabh Portuguese: @doryhelioaires Urdu: Momini0 These animations are largely made using a custom Python library, manim. See the FAQ comments here: https://3b1b.co/faq#manim https://github.com/3b1b/manim https://github.com/ManimCommunity/manim/ All code for specific videos is visible here: https://github.com/3b1b/videos/ The music is by Vincent Rubinetti. https://www.vincentrubinetti.com https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u 3blue1brown is a channel about animating math, in all senses of the word animate. If youre reading the bottom of a video description, Im guessing youre more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly. Mailing list: https://3blue1brown.substack.com Twitter: https://twitter.com/3blue1brown Instagram: https://www.instagram.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Facebook: https://www.facebook.com/3blue1brown Patreon: https://patreon.com/3blue1brown Website: https://www.3blue1brown.com$

$Beyond the Mandelbrot set, an intro to holomorphic dynamics An intro to holomorphic dynamics, the study of iterated complex functions. Video on Newtons fractal: https://youtu.be/-RdOwhmqP5s Special thanks to these supporters: https://3b1b.co/lessons/holomorphic-dynamics#thanks Extra special thanks to Sergey Shemyakov, of Aix-Marseille University, for helpful conversations and for introducing me to this phenomenon. Introduction to Fatou sets and Julia sets, including a discussion of Montels theorem and its consequences: http://www.math.stonybrook.edu/~scott/Papers/India/Fatou-Julia.pdf Numberphile with Ben Sparks on the Mandelbrot set: https://youtu.be/FFftmWSzgmk Ben explains how he made the Geogebra files on his channel here: https://youtu.be/ICqj7nJbiRI (part 1) https://youtu.be/3BXoNOahFJk (part 2) Excellent article on Acko.net, from the basics of building up complex numbers to Julia sets. https://acko.net/blog/how-to-fold-a-julia-fractal/ Bit of a side note, but if you want an exceedingly beautiful rendering of the quaternion-version of Julia fractals, take a look at this Inigo Quilez video: https://www.youtube.com/watch?v=rQ2bnU4dkso I first saw Fatous theorem in this article: https://projecteuclid.org/journals/communications-in-mathematical-physics/volume-91/issue-2/On-the-iteration-of-a-rational-function computer-experiments/cmp/1103940533.pdf Moduli spaces of Newton maps: https://arxiv.org/pdf/1512.05098.pdf On Montels theorem: https://people.ucsc.edu/~fmonard/Sp17_Math207/lecture11.pdf On Newtons Fractal: https://core.ac.uk/download/pdf/1633889.pdf Thanks to these viewers for their contributions to translations Hebrew: Omer Tuchfeld These animations are largely made using a custom python library, manim. See the FAQ comments here: https://www.3blue1brown.com/faq#manim https://github.com/3b1b/manim https://github.com/ManimCommunity/manim/ You can find code for specific videos and projects here: https://github.com/3b1b/videos/ Music by Vincent Rubinetti. https://www.vincentrubinetti.com/ Download the music on Bandcamp: https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown Stream the music on Spotify: https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u Timestamps: 0:00 - Intro 3:02 - Rational functions 4:15 - The Mandelbrot set 8:12 - Fixed points and stability 12:51 - Cycles 16:25 - Hidden Mandelbrot 21:17 - Fatou sets and Julia sets 26:24 - Final thoughts 3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe: http://3b1b.co/subscribe Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Instagram: https://www.instagram.com/3blue1brown_animations/ Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown$