The worlds smallest transformer model @SerranoAcademy
The worlds smallest transformer model  @SerranoAcademy
Uploaded July 2026 | Updated November 2026, 3 hours ago
This is the second of a trilogy of videos titled "The world's smallest transformer model".

In this video, we build a tiny transformer model with one hidden layer of 2 nodes, and no bias or activation function. This transformer is able to form small sentences based on a dataset of two sentences.

Video 1: How neural networks bend space youtube.com/watch?v=3THzlSIzfug
Video 2: The world's smallest transformer model (this one)
Video 3: The world's smallest attention mechanism youtube.com/watch?v=J4hSc9He614
ADDENDUM: Key and Query Matrices worked out in detail youtube.com/watch?v=i7VNOdvdrCA

The codelab is here
github.com/luisguiserrano/tiny_llm

Get the second edition of Grokking Machine Learning!
manning.com/books/grokking-machine-learning
Use this code for a 50% discount! Luis50

00:00 Introduction
00:45 Summary of the previous video
03:59 The dataset and the model
9:10 The architecture
10:27 The understander
12:43 Dimensions, overfitting, and underfitting
15:34 The talker
21:15 Point system (Dot product)
26:11 Softmax and probabilities
31:02 Temperature
33:22 Adding more data
37:33 Why do we need attention?
The worlds smallest transformer modelDiscrete Dynamical Systems - Eigenvalues and EigenvectorsA Friendly Introduction to Generative Adversarial Networks (GANs)Model that won the 2024 Physics Nobel Prize - Hopfield NetworksJosh Starmer and Luis Serrano livestream 2 - Double BAM!Shannon Entropy and Information GainHappy 2025, and thank you for your support!What is AdaBoost? Friendly explanation with code!A friendly introduction to Deep Learning and Neural NetworksTraining Latent Dirichlet Allocation: Gibbs Sampling (Part 2 of 2)What is the best approximation to Pi?Why is KL Divergence not symmetric?
Luis Serrano Academy |

The world's smallest transformer model

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER