How do Transformer Models keep track of the order of words? Positional Encoding @SerranoAcademy
How do Transformer Models keep track of the order of words? Positional Encoding  @SerranoAcademy
Uploaded November 2024 | Updated November 2026, 3 days ago
Transformer models can generate language really well, but how do they do it?
A very important step of the pipeline is the Positional Encoding step, which injects the order of the words into the LLM, in order for it to tell the difference between sentences with the same words, but in different order.
This is part of a series of videos on Large Language Models:
Video 1: The attention mechanism youtube.com/watch?v=OxCpWwDCDFQ
Video 2: The math behind the attention mechanism youtube.com/watch?v=UPtG_38Oq8o
Video 3: Transformer models youtube.com/watch?v=qaWMOYf4ri8

Grokking Machine Learning Book:
manning.com/books/grokking-machine-learning
40% discount promo code: serranoyt

00:00 Introduction
03:01 How much to add to each word?
03:29 Some trigonometry
06:20 The formulas
How do Transformer Models keep track of the order of words? Positional EncodingA Friendly Introduction to Machine LearningThe worlds smallest attention mechanismStable Diffusion - How to build amazing images with AITalk about AI Agents live with Miguel Otero, Josh Starmer, Luis Serrano @TheNeuralMaze @statquestSupport Vector Machines (SVMs): A friendly introductionYou are much better at math than you thinkThe three steps to make a reliable chatbot: Preamble, Fine-tuning, and RAGGeometric series and my Irish heritageAMA with Luis SerranoThe Attention Mechanism in Large Language ModelsDeep Neural Networks - USF Data Science Seminar by Luis Serrano
Luis Serrano Academy |

How do Transformer Models keep track of the order of words? Positional Encoding

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER