Uploaded August 2021 | Updated September 2026, 1 week ago
π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY:
Discord βΊ discord.gg/peBrCpheKE
π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack βΊ aiepiphany.substack.com
β€οΈ Become The AI Epiphany Patreon β€οΈ βΊ patreon.com/theaiepiphany
In this video I cover ALiBi model from the "Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation" paper.
Instead of using positional embeddings (like e.g. the original transformer) they added non-learnable biases directly into the query-key matrix and achieved extraordinary extrapolation results (i.e. great perf on longer sequences).
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β Paper: ofir.io/train_short_test_long.pdf
β Code: github.com/ofirpress/attention_with_linear_biases
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00 Intro
02:00 Main results
03:30 Time and memory tradeoffs
05:00 ALiBi method explained
10:00 Results, low data regime
12:35 Generalization to different datasets
13:15 Results, big data regime
16:00 Why does it work and future work
21:10 Outro
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π° BECOME A PATREON OF THE AI EPIPHANY β€οΈ
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany βΊ patreon.com/theaiepiphany
One-time donation:
paypal.com/paypalme/theaiepiphany
Much love! β€οΈ
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar VeliΔkoviΔ
Zvonimir Sabljic
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π‘ The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition".
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π CONNECT WITH ME ON SOCIAL
LinkedIn βΊ linkedin.com/in/aleksagordic
Twitter βΊ twitter.com/gordic_aleksa
Instagram βΊ instagram.com/aiepiphany
Facebook βΊ facebook.com/aiepiphany
π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY:
Discord βΊ discord.gg/peBrCpheKE
π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack βΊ aiepiphany.substack.com
π» FOLLOW ME ON GITHUB FOR ML PROJECTS:
GitHub βΊ github.com/gordicaleksa
π FOLLOW ME ON MEDIUM:
Medium βΊ gordicaleksa.medium.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#alibi #transformers #extrapolation
π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY:
Discord βΊ discord.gg/peBrCpheKE
π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack βΊ aiepiphany.substack.com
β€οΈ Become The AI Epiphany Patreon β€οΈ βΊ patreon.com/theaiepiphany
In this video I cover ALiBi model from the "Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation" paper.
Instead of using positional embeddings (like e.g. the original transformer) they added non-learnable biases directly into the query-key matrix and achieved extraordinary extrapolation results (i.e. great perf on longer sequences).
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
β Paper: ofir.io/train_short_test_long.pdf
β Code: github.com/ofirpress/attention_with_linear_biases
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
βοΈ Timetable:
00:00 Intro
02:00 Main results
03:30 Time and memory tradeoffs
05:00 ALiBi method explained
10:00 Results, low data regime
12:35 Generalization to different datasets
13:15 Results, big data regime
16:00 Why does it work and future work
21:10 Outro
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π° BECOME A PATREON OF THE AI EPIPHANY β€οΈ
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany βΊ patreon.com/theaiepiphany
One-time donation:
paypal.com/paypalme/theaiepiphany
Much love! β€οΈ
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar VeliΔkoviΔ
Zvonimir Sabljic
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π‘ The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition".
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
π CONNECT WITH ME ON SOCIAL
LinkedIn βΊ linkedin.com/in/aleksagordic
Twitter βΊ twitter.com/gordic_aleksa
Instagram βΊ instagram.com/aiepiphany
Facebook βΊ facebook.com/aiepiphany
π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY:
Discord βΊ discord.gg/peBrCpheKE
π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER:
Substack βΊ aiepiphany.substack.com
π» FOLLOW ME ON GITHUB FOR ML PROJECTS:
GitHub βΊ github.com/gordicaleksa
π FOLLOW ME ON MEDIUM:
Medium βΊ gordicaleksa.medium.com
β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬
#alibi #transformers #extrapolation
