Uploaded September 2022 | Updated September 2026, 1 week ago
β€οΈ Become The AI Epiphany Patreon β€οΈ
patreon.com/theaiepiphany
π¨βπ©βπ§βπ¦ Join our Discord community π¨βπ©βπ§βπ¦
discord.gg/peBrCpheKE
In this 4th video of the Large Language Model series I walk you through the BigScience's BLOOM model codebase!
The main focus is on understanding the 3D parallelism:
* Pipeline parallelism
* Model parallelism
* Data parallelism
A set of beautiful engineering ideas that are behind all of the recent scaling efforts and ML success stories!
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β BLOOM code: github.com/bigscience-workshop/Megatron-DeepSpeed
β Ultimate Guide to Scaling Video: youtube.com/watch?v=hc0u4avAkuM
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00:00 Intro - focusing on the 3D parallelism!
00:02:00 Quick setup
00:05:00 Stepping through the eval script
00:11:13 3D paralellism - model construction
00:15:00 Sharding the embedding table (model parallelism)
00:20:09 Sharding the transformer layer
00:22:30 LayerNorm fused kernels
00:23:50 Sharding the attention layer
00:25:15 ColumnParallel and RowParallel sharding
00:31:30 Synchronizing input and output embedding tables
00:34:45 Building the dataset (data parallelism)
00:39:15 3D parallelism - forward pass
00:39:25 Pipeline parallelism communication
00:43:35 Pass through the sharded embedding table
00:52:15 Pass through the sharded transformer layer
01:01:36 Sharded logit and cross-entropy computation
01:05:30 Recap
01:11:15 Outro
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π° BECOME A PATREON OF THE AI EPIPHANY β€οΈ
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany - patreon.com/theaiepiphany
One-time donation - paypal.com/paypalme/theaiepiphany
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar VeliΔkoviΔ
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
πΌ LinkedIn - linkedin.com/in/aleksagordic
π¦ Twitter - twitter.com/gordic_aleksa
π¨βπ©βπ§βπ¦ Discord - discord.gg/peBrCpheKE
πΊ YouTube - youtube.com/c/TheAIEpiphany
π Medium - gordicaleksa.medium.com
π» GitHub - github.com/gordicaleksa
π’ AI Newsletter - aiepiphany.substack.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#bloom #bigscience #3dparallelism #scaling
β€οΈ Become The AI Epiphany Patreon β€οΈ
patreon.com/theaiepiphany
π¨βπ©βπ§βπ¦ Join our Discord community π¨βπ©βπ§βπ¦
discord.gg/peBrCpheKE
In this 4th video of the Large Language Model series I walk you through the BigScience's BLOOM model codebase!
The main focus is on understanding the 3D parallelism:
* Pipeline parallelism
* Model parallelism
* Data parallelism
A set of beautiful engineering ideas that are behind all of the recent scaling efforts and ML success stories!
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β BLOOM code: github.com/bigscience-workshop/Megatron-DeepSpeed
β Ultimate Guide to Scaling Video: youtube.com/watch?v=hc0u4avAkuM
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00:00 Intro - focusing on the 3D parallelism!
00:02:00 Quick setup
00:05:00 Stepping through the eval script
00:11:13 3D paralellism - model construction
00:15:00 Sharding the embedding table (model parallelism)
00:20:09 Sharding the transformer layer
00:22:30 LayerNorm fused kernels
00:23:50 Sharding the attention layer
00:25:15 ColumnParallel and RowParallel sharding
00:31:30 Synchronizing input and output embedding tables
00:34:45 Building the dataset (data parallelism)
00:39:15 3D parallelism - forward pass
00:39:25 Pipeline parallelism communication
00:43:35 Pass through the sharded embedding table
00:52:15 Pass through the sharded transformer layer
01:01:36 Sharded logit and cross-entropy computation
01:05:30 Recap
01:11:15 Outro
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π° BECOME A PATREON OF THE AI EPIPHANY β€οΈ
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany - patreon.com/theaiepiphany
One-time donation - paypal.com/paypalme/theaiepiphany
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar VeliΔkoviΔ
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
πΌ LinkedIn - linkedin.com/in/aleksagordic
π¦ Twitter - twitter.com/gordic_aleksa
π¨βπ©βπ§βπ¦ Discord - discord.gg/peBrCpheKE
πΊ YouTube - youtube.com/c/TheAIEpiphany
π Medium - gordicaleksa.medium.com
π» GitHub - github.com/gordicaleksa
π’ AI Newsletter - aiepiphany.substack.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#bloom #bigscience #3dparallelism #scaling
