Uploaded October 2026 | Updated November 2026, 2 days ago
Asked to repeat one word, the model started saying strange things. The reason is often not the model but the tokenizer. Four habits that change what a model understands, in one video.
We read text. The model only reads tokens. The gap between the two is the root of many strange mistakes.
- Habit 1, the space is part of the token: in GPT-style tokenizers the space before a word sticks to that word's token, so "hello" and " hello" are two different IDs. A prompt that ends with an extra space leaves the model somewhere it rarely saw in training, and the answer can come out strange
- Habit 2, capital letters: "Hello", "hello" and "HELLO" are three different tokens, and all-caps words often split into more pieces. The model has to learn the link between them on its own
- Habit 3, numbers: GPT-4's tokenizer cuts digits into chunks of three, so a number like 12345 becomes 123 and 45. The model cannot clearly see the ones and the tens, which is one reason math on big numbers goes wrong. Llama 2 does the opposite: every digit is its own token
- Habit 4, glitch tokens: in the GPT-2 and GPT-3 vocabulary a Reddit username, SolidGoldMagikarp, became a whole token. It was common in the tokenizer's data but almost absent from the model's training, so its embedding was never properly learned. Asked to repeat it, the model started saying strange things
- The catch: you cannot see these mistakes by reading the prompt, because you read text, not tokens. When a model acts odd, paste your prompt into a tokenizer playground and look at the pieces. The trailing space and the oddly cut numbers show up right there
- Illustrative on screen: the odd reply text in the opening, the token IDs shown for hello (15339), " hello" (24748), Hello (9906) and HELLO (51812), the HE / LL / O split, the 12345 + 1 = 12335 answer, the "seen in training" and embedding bars, the playground window (pieces hello, space, 123, 45, space; 5 tokens, 12 characters), the Reddit card (drawn by hand, not the official logo) and the comment text
Coming up next in LLM Interview: why can't LLMs count the letters in a word? Have you ever caught a bug caused by a trailing space? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Tokenizer #Tokenization #GPT4 #SolidGoldMagikarp #GlitchTokens #BPE #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie
Asked to repeat one word, the model started saying strange things. The reason is often not the model but the tokenizer. Four habits that change what a model understands, in one video.
We read text. The model only reads tokens. The gap between the two is the root of many strange mistakes.
- Habit 1, the space is part of the token: in GPT-style tokenizers the space before a word sticks to that word's token, so "hello" and " hello" are two different IDs. A prompt that ends with an extra space leaves the model somewhere it rarely saw in training, and the answer can come out strange
- Habit 2, capital letters: "Hello", "hello" and "HELLO" are three different tokens, and all-caps words often split into more pieces. The model has to learn the link between them on its own
- Habit 3, numbers: GPT-4's tokenizer cuts digits into chunks of three, so a number like 12345 becomes 123 and 45. The model cannot clearly see the ones and the tens, which is one reason math on big numbers goes wrong. Llama 2 does the opposite: every digit is its own token
- Habit 4, glitch tokens: in the GPT-2 and GPT-3 vocabulary a Reddit username, SolidGoldMagikarp, became a whole token. It was common in the tokenizer's data but almost absent from the model's training, so its embedding was never properly learned. Asked to repeat it, the model started saying strange things
- The catch: you cannot see these mistakes by reading the prompt, because you read text, not tokens. When a model acts odd, paste your prompt into a tokenizer playground and look at the pieces. The trailing space and the oddly cut numbers show up right there
- Illustrative on screen: the odd reply text in the opening, the token IDs shown for hello (15339), " hello" (24748), Hello (9906) and HELLO (51812), the HE / LL / O split, the 12345 + 1 = 12335 answer, the "seen in training" and embedding bars, the playground window (pieces hello, space, 123, 45, space; 5 tokens, 12 characters), the Reddit card (drawn by hand, not the official logo) and the comment text
Coming up next in LLM Interview: why can't LLMs count the letters in a word? Have you ever caught a bug caused by a trailing space? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Tokenizer #Tokenization #GPT4 #SolidGoldMagikarp #GlitchTokens #BPE #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie










