The worlds smallest attention mechanism @SerranoAcademy
The worlds smallest attention mechanism  @SerranoAcademy
Uploaded August 2026 | Updated November 2026, 3 hours ago
This is the third and last video of a trilogy of videos titled "The world's smallest transformer model".

In this video, we add an attention mechanism to the tiny transformer we created in the second video, that is trained in a dataset of 2 sentences and 4 words, and with a 4-2-4 architecture. This attention mechanism allows the network to carry memory of the previous words in the context. The attention mechanism is analogous to a gravitational force that pulls words together. When ambiguities arise, multi-head attention comes to the rescue.

Video 1: How neural networks bend space youtube.com/watch?v=3THzlSIzfug
Video 2: The world's smallest transformer model youtube.com/watch?v=3ZxbaeqZIHQ
Video 3: The world's smallest attention mechanism (This one!)
ADDENDUM: Key and Query Matrices worked out in detail youtube.com/watch?v=i7VNOdvdrCA

The codelab is here
github.com/luisguiserrano/tiny_llm

Get the second edition of Grokking Machine Learning!
manning.com/books/grokking-machine-learning
Use this code for a 50% discount! Luis50

00:00 Introduction
00:35 Recap of previous video
05:38 The attention mechanism
07:54 Word gravity
12:14 The Value matrix
18:42 The attention formula
20:58 Similarity
25:37 The Key and Query matrices
34:25 Why do we need K and Q?
40:33 Softmax
44:21 Summary of attention
45:52 Multi-head attention
46:57 Full transformer architecture
The worlds smallest attention mechanismStable Diffusion - How to build amazing images with AITalk about AI Agents live with Miguel Otero, Josh Starmer, Luis Serrano @TheNeuralMaze @statquestSupport Vector Machines (SVMs): A friendly introductionYou are much better at math than you thinkThe three steps to make a reliable chatbot: Preamble, Fine-tuning, and RAGGeometric series and my Irish heritageAMA with Luis SerranoThe Attention Mechanism in Large Language ModelsDeep Neural Networks - USF Data Science Seminar by Luis SerranoNaive Bayes classifier: A friendly approachClustering: K-means and Hierarchical
Luis Serrano Academy |

The world's smallest attention mechanism

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER