Uploaded March 2022 | Updated September 2026, 1 week ago
π¨βπ©βπ§βπ¦ Join our Discord community π¨βπ©βπ§βπ¦
discord.gg/peBrCpheKE
In this video I cover "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (ΞΌTransfer)" paper that makes optimal hyperparameters stable w.r.t. width scaling!
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β Paper: arxiv.org/abs/2203.03466
β Their previous paper: arxiv.org/abs/2011.14522
β ΞΌTransfer tool: github.com/microsoft/mup
β DeepNet paper: arxiv.org/abs/2203.00555
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00 uTransfer introduced
04:30 Previous work (tensor programs IV)
10:00 NTK - neural tangent kernel recap
16:30 abc parametrization
19:25 How does learning happen in NTK?
33:50 Connections to Central Limit Theorem
39:30 Maximal Update Parametrization in Practice
45:50 DeepNet paper connection
48:20 Results (width is all you need?)
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
THE AI EPIPHANY PATREONS WALL β€οΈ
Eli Mahler
Kevin Stone
Petar VeliΔkoviΔ
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
πΌ LinkedIn - linkedin.com/in/aleksagordic
π¦ Twitter - twitter.com/gordic_aleksa
π¨βπ©βπ§βπ¦ Discord - discord.gg/peBrCpheKE
πΊ YouTube - youtube.com/c/TheAIEpiphany
π Medium - gordicaleksa.medium.com
π» GitHub - github.com/gordicaleksa
π’ AI Newsletter - aiepiphany.substack.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#ΞΌTransfer #MaximalUpdateParametrization #scalingl
π¨βπ©βπ§βπ¦ Join our Discord community π¨βπ©βπ§βπ¦
discord.gg/peBrCpheKE
In this video I cover "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (ΞΌTransfer)" paper that makes optimal hyperparameters stable w.r.t. width scaling!
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β Paper: arxiv.org/abs/2203.03466
β Their previous paper: arxiv.org/abs/2011.14522
β ΞΌTransfer tool: github.com/microsoft/mup
β DeepNet paper: arxiv.org/abs/2203.00555
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00 uTransfer introduced
04:30 Previous work (tensor programs IV)
10:00 NTK - neural tangent kernel recap
16:30 abc parametrization
19:25 How does learning happen in NTK?
33:50 Connections to Central Limit Theorem
39:30 Maximal Update Parametrization in Practice
45:50 DeepNet paper connection
48:20 Results (width is all you need?)
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
THE AI EPIPHANY PATREONS WALL β€οΈ
Eli Mahler
Kevin Stone
Petar VeliΔkoviΔ
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
πΌ LinkedIn - linkedin.com/in/aleksagordic
π¦ Twitter - twitter.com/gordic_aleksa
π¨βπ©βπ§βπ¦ Discord - discord.gg/peBrCpheKE
πΊ YouTube - youtube.com/c/TheAIEpiphany
π Medium - gordicaleksa.medium.com
π» GitHub - github.com/gordicaleksa
π’ AI Newsletter - aiepiphany.substack.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#ΞΌTransfer #MaximalUpdateParametrization #scalingl
