Evolution of back propagation: Who invented it? @CodeEmporium
Evolution of back propagation: Who invented it?  @CodeEmporium
Uploaded March 2025 | Updated November 2026, 3 hours ago
Let's understand the evolution of back propagation. Who created it?

Last video on multiplayer perceptron: youtu.be/poNXEd3jCwI

RESOURCES
[1 ๐Ÿ“š] 1847, Cauchy invented gradient descent. Here is the English rendition of this work with comments: kurims.kyoto-u.ac.jp/EMIS/journals/DMJDMV/vol-ismp/40_lemarechal-claude.pdf
[2 ๐Ÿ“š] Hinton explaining the mechanics of back propagation introduced by Rumelhart, Williams and himself: youtube.com/watch?v=VCT1N0EsGj0
[3 ๐Ÿ“š] Proof that gradient vector points in the direction of steepest ascent: sootlasten.github.io/2017/gradient-steepest-ascent
[4 ๐Ÿ“š] Online 3D plotter: c3d.libretexts.org/CalcPlot3D/index.html
[5 ๐Ÿ“š] 1986 paper that is the most famous implementation of back propagation: semanticscholar.org/paper/Learning-representations-by-back-propagating-errors-Rumelhart-Hinton/052b1d8ce63b07fec3de9dbb583772d860b7c769
[6 ๐Ÿ“š] Henry J Kellyโ€™s 1960 paper on determining the Optimal flight path and hence developing their own version of back propagation: gwern.net/doc/statistics/decision/1960-kelley.pdf
[7 ๐Ÿ“š] Backstory of back propagation: yuxi-liu-wired.github.io/essays/posts/backstory-of-backpropagation
[8 ๐Ÿ“š] Tourist guide to differential equations by 3blue1brown: youtube.com/watch?v=p_di4Zn4wz4
[9 ๐Ÿ“š] Khan Academy on derivatives, directional derivatives: khanacademy.org/math/multivariable-calculus/multivariable-derivatives/gradient-and-directional-derivatives/v/gradient
[10 ๐Ÿ“š] A nice tutorial on reverse mode automatic differentiation: rufflewind.com/2016-12-30/reverse-mode-automatic-differentiation
[11 ๐Ÿ“š ] Everything you need to know about control theory. Amazing resource: youtu.be/lBC1nEq0_nk?si=-Jkh7PZl0bjY6LGE
[12 ๐Ÿ“š] Thesis by Paul Werbos in 1974 (one of the first successful implementations of a back propagation learning procedure on a multi-layered perceptron): gwern.net/doc/ai/nn/1974-werbos.pdf
[13 ๐Ÿ“š] How the adjoint method is similar to the reverse-mode Automatic differentiation: github.com/fancompute/workshop-invdesign/issues/3
[14 ๐Ÿ“š] The 1969 book on optimal control by Bryson and Ho: http://e.guigon.free.fr/rsc/book/BrysonHo75.pdf
[15 ๐Ÿ“š ] The adjoint method of gradient calculation: youtube.com/watch?v=Yiz92Ekn7vU
[16 ๐Ÿ“š] Nice sheet on method of Lagrange multipliers (this is used in the adjoint state method): https://www.cs.toronto.edu/~fleet/courses/C11/Handouts/LagrangeMultipliers.pdf

ABOUT ME
โญ• Subscribe: youtube.com/c/CodeEmporium?sub_confirmation=1
๐Ÿ“š Medium Blog: medium.com/@dataemporium
๐Ÿ’ป Github: github.com/ajhalthor
๐Ÿ‘” LinkedIn: linkedin.com/in/ajay-halthor-477974bb

PLAYLISTS FROM MY CHANNEL
โญ• Deep Learning 101: youtube.com/playlist?list=PLTl9hO2Oobd_NwyY_PeSYrYfsvHZnHGPU
โญ• Natural Language Processing 101: youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE
โญ• Reinforcement Learning 101: youtube.com/playlist?list=PLTl9hO2Oobd9kS--NgVz0EPNyEmygV1Ha&si=AuThDZJwG19cgTA8
Natural Language Processing 101: youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE&si=LsVy8RDPu8jeO-cc
โญ• Transformers from Scratch: youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE
โญ• ChatGPT Playlist: youtube.com/playlist?list=PLTl9hO2Oobd9coYT6XsTraTBo4pL1j4HJ

MATH COURSES (7 day free trial)
๐Ÿ“• Mathematics for Machine Learning: imp.i384100.net/MathML
๐Ÿ“• Calculus: imp.i384100.net/Calculus
๐Ÿ“• Statistics for Data Science: imp.i384100.net/AdvancedStatistics
๐Ÿ“• Bayesian Statistics: imp.i384100.net/BayesianStatistics
๐Ÿ“• Linear Algebra: imp.i384100.net/LinearAlgebra
๐Ÿ“• Probability: imp.i384100.net/Probability

OTHER RELATED COURSES (7 day free trial)
๐Ÿ“• โญ Deep Learning Specialization: imp.i384100.net/Deep-Learning
๐Ÿ“• Python for Everybody: imp.i384100.net/python
๐Ÿ“• MLOps Course: imp.i384100.net/MLOps
๐Ÿ“• Natural Language Processing (NLP): imp.i384100.net/NLP
๐Ÿ“• Machine Learning in Production: imp.i384100.net/MLProduction
๐Ÿ“• Data Science Specialization: imp.i384100.net/DataScience
๐Ÿ“• Tensorflow: imp.i384100.net/Tensorflow

CHAPTERS
0:00 Why perceptron convergence learning could no longer be used
2:00 Why Analytical methods are infeasible
4:00 How numerical methods like gradient descent are feasible
7:00 Gradient vector and direction of steepest ascent
13:58 An efficient way to compute gradients
17:32 What is "backward" in back propagation and why
18:48 How this efficient gradient compute method is used in neural networks
20:08 Who invented backprop
23:22 Quiz Time
24:37 Summary
Evolution of back propagation: Who invented it?Region Proposals - Explained!Time Series Modeling   trend and seasonality #deeplearning #machinelearningQ-learning - Explained!NLP with Neural Networks | ngram to LLMsFeed forward networks in TransformersPlaylist to Code Transformer ModelFull Timeline of Computer Vision & Neural NetworksMutli head attention for TransformerRAG - Explained!How Transformers Learned to See at Every ScaleHow much training data does a neural network need?
CodeEmporium |

Evolution of back propagation: Who invented it?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER