Uploaded June 2019 | Updated September 2026, 2 weeks ago
I had many unsleep nights to get the point how most of the popular Deep Learning Optimization Algorithms are working, how to compare those ones and what is the intuition behind them avoiding very complex math notes.
This is how this video had born.
In this video I analyze all mentioned techniques one after one starting from Momentum, then Rmsprop and finally introduce Adam that is the combination of SGD with Momentum and Rmsprop, as theory says.
Some definitions what is what:
- Stochatic Gradient Descent with Momentum : 0:55
- rmsprop (Root Mean Square Propagation) : 2:31
- Adam (Adaptive moment estimation) : 5:31
- Comparison of SGD and rmsprop and Adam: 7:09
- History of Deep Learning Optimization Algorithms: 13:04
Some useful references:
- Gentle Introduction to the Adam Optimization Algorithm for Deep Learning by Jason Brownlee ( machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning)
- Adam — latest trends in deep learning optimization by Vitaly Bushaev (towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c)
- Intro to optimization in deep learning: Momentum, RMSProp and Adam by Ayoosh Kathuria (blog.paperspace.com/intro-to-optimization-momentum-rmsprop-adam/)
- and of course Deep Learning Specialization on Coursera by Andrew Ng.
I hope that this video will be useful for Deep Learning specialists, Machine Learning Engineers, Data Scientists, Data Analysts who want to learn more than average enthusiast.
#deeplearning #rmsprop #adamoptimizer
I had many unsleep nights to get the point how most of the popular Deep Learning Optimization Algorithms are working, how to compare those ones and what is the intuition behind them avoiding very complex math notes.
This is how this video had born.
In this video I analyze all mentioned techniques one after one starting from Momentum, then Rmsprop and finally introduce Adam that is the combination of SGD with Momentum and Rmsprop, as theory says.
Some definitions what is what:
- Stochatic Gradient Descent with Momentum : 0:55
- rmsprop (Root Mean Square Propagation) : 2:31
- Adam (Adaptive moment estimation) : 5:31
- Comparison of SGD and rmsprop and Adam: 7:09
- History of Deep Learning Optimization Algorithms: 13:04
Some useful references:
- Gentle Introduction to the Adam Optimization Algorithm for Deep Learning by Jason Brownlee ( machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning)
- Adam — latest trends in deep learning optimization by Vitaly Bushaev (towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c)
- Intro to optimization in deep learning: Momentum, RMSProp and Adam by Ayoosh Kathuria (blog.paperspace.com/intro-to-optimization-momentum-rmsprop-adam/)
- and of course Deep Learning Specialization on Coursera by Andrew Ng.
I hope that this video will be useful for Deep Learning specialists, Machine Learning Engineers, Data Scientists, Data Analysts who want to learn more than average enthusiast.
#deeplearning #rmsprop #adamoptimizer
