Uploaded November 2026 | Updated November 2026, 2 days ago
Shuffle the words of a sentence, and plain attention doesn't notice a thing. Here is why word order matters, and how transformers put it back.
- "Dog bites man" and "man bites dog": the same words, the opposite meaning
- Step 1, attention can't see order: every token scores every other token from the dot product of a query and a key, and nothing in that formula says which token sits where. Shuffle the input and the output just shuffles the same way (permutation equivariance). To plain self-attention, both sentences are the same bag of words
- Step 2, position as a vector, added to the token's embedding. The original transformer (2017) used sine and cosine waves at different frequencies, like the hands of a clock: no parameters, and every position gets its own pattern
- Step 3, learned positions: BERT and GPT-2 learn a table with one row per position, just like token embeddings, updated by training with all the other weights
- Step 4, RoPE (rotary position embedding), used by most open LLMs today, like Llama: it adds nothing to the embedding, it rotates the query and key in every attention layer by an angle that depends on the position, so the score sees only the distance between two tokens, not where they sit
- The catch: a learned table has a fixed number of rows. GPT-2's has 1,024 positions, so it can't take a longer input. And position is added, not concatenated: in GPT-2 small both vectors have 768 dimensions and the sum is still 768. The model learns to read the two signals apart
- Illustrative on screen: the score-grid colours, the vector bars and wave shapes, the "pos 1…6" table rows and their cell colours, the clock times and mini clocks, the token-table words (the, dog, bites, man, cat, runs), the 8-word example sentence "the dog bites the man in the park" and its token pairs, the rotation angles on the dials, the score bar lengths, the "tok" input strip, and the comment text and slider in the end scene
Next is an LLM Interview reel: why do transformers need positional encoding? How much does word order matter in your language? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Transformers #PositionalEncoding #RoPE #SelfAttention #Attention #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie
Shuffle the words of a sentence, and plain attention doesn't notice a thing. Here is why word order matters, and how transformers put it back.
- "Dog bites man" and "man bites dog": the same words, the opposite meaning
- Step 1, attention can't see order: every token scores every other token from the dot product of a query and a key, and nothing in that formula says which token sits where. Shuffle the input and the output just shuffles the same way (permutation equivariance). To plain self-attention, both sentences are the same bag of words
- Step 2, position as a vector, added to the token's embedding. The original transformer (2017) used sine and cosine waves at different frequencies, like the hands of a clock: no parameters, and every position gets its own pattern
- Step 3, learned positions: BERT and GPT-2 learn a table with one row per position, just like token embeddings, updated by training with all the other weights
- Step 4, RoPE (rotary position embedding), used by most open LLMs today, like Llama: it adds nothing to the embedding, it rotates the query and key in every attention layer by an angle that depends on the position, so the score sees only the distance between two tokens, not where they sit
- The catch: a learned table has a fixed number of rows. GPT-2's has 1,024 positions, so it can't take a longer input. And position is added, not concatenated: in GPT-2 small both vectors have 768 dimensions and the sum is still 768. The model learns to read the two signals apart
- Illustrative on screen: the score-grid colours, the vector bars and wave shapes, the "pos 1…6" table rows and their cell colours, the clock times and mini clocks, the token-table words (the, dog, bites, man, cat, runs), the 8-word example sentence "the dog bites the man in the park" and its token pairs, the rotation angles on the dials, the score bar lengths, the "tok" input strip, and the comment text and slider in the end scene
Next is an LLM Interview reel: why do transformers need positional encoding? How much does word order matter in your language? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Transformers #PositionalEncoding #RoPE #SelfAttention #Attention #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie










