Tokenization in LLM: 4 Quirks That Confuse Models @KGPTalkie
Tokenization in LLM: 4 Quirks That Confuse Models  @KGPTalkie
Uploaded October 2026 | Updated November 2026, 2 days ago
Asked to repeat one word, the model started saying strange things. The reason is often not the model but the tokenizer. Four habits that change what a model understands, in one video.

We read text. The model only reads tokens. The gap between the two is the root of many strange mistakes.

- Habit 1, the space is part of the token: in GPT-style tokenizers the space before a word sticks to that word's token, so "hello" and " hello" are two different IDs. A prompt that ends with an extra space leaves the model somewhere it rarely saw in training, and the answer can come out strange
- Habit 2, capital letters: "Hello", "hello" and "HELLO" are three different tokens, and all-caps words often split into more pieces. The model has to learn the link between them on its own
- Habit 3, numbers: GPT-4's tokenizer cuts digits into chunks of three, so a number like 12345 becomes 123 and 45. The model cannot clearly see the ones and the tens, which is one reason math on big numbers goes wrong. Llama 2 does the opposite: every digit is its own token
- Habit 4, glitch tokens: in the GPT-2 and GPT-3 vocabulary a Reddit username, SolidGoldMagikarp, became a whole token. It was common in the tokenizer's data but almost absent from the model's training, so its embedding was never properly learned. Asked to repeat it, the model started saying strange things
- The catch: you cannot see these mistakes by reading the prompt, because you read text, not tokens. When a model acts odd, paste your prompt into a tokenizer playground and look at the pieces. The trailing space and the oddly cut numbers show up right there
- Illustrative on screen: the odd reply text in the opening, the token IDs shown for hello (15339), " hello" (24748), Hello (9906) and HELLO (51812), the HE / LL / O split, the 12345 + 1 = 12335 answer, the "seen in training" and embedding bars, the playground window (pieces hello, space, 123, 45, space; 5 tokens, 12 characters), the Reddit card (drawn by hand, not the official logo) and the comment text

Coming up next in LLM Interview: why can't LLMs count the letters in a word? Have you ever caught a bug caused by a trailing space? Comment and tell me. Follow for the LLM Concepts series.

kgptalkie.com

#shorts #LLM #Tokenizer #Tokenization #GPT4 #SolidGoldMagikarp #GlitchTokens #BPE #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie
Tokenization in LLM: 4 Quirks That Confuse ModelsVocabulary Size vs Context Window: Why 128K Means Two Things | LLM ConceptsQwen 3.8 on Ollama vs Muse Glimmer vs Gemma 4 - Local LLM Coding Test on OllamaI Ran Qwen 3.8 Flash Next (Qwen 4) on CPU OnlyInstagram Feed System Design: Interview Question ExplainedProduction RAG for Finance with Qwen 3.8 27B: Fully Local Ollama + Qdrant (Live)AI Chatbot Hallucination: The Bot Promised the Wrong Refund | AI System Design InterviewSentencePiece Tokenizer Explained | Tokenization Without SpacesProduction RAG for Finance with NVIDIA Nemotron 3.5 Lightning: Fully Local Ollama + Qdrant (Live)Ryzen AI Halo vs DGX Spark: $700 Apart, Almost the Same Speed #shortsFine-Tune Gemma 3 270M on MacBook with Unsloth Studio (No Code)Hindi vs English Tokens: Why Hindi Needs 3x More
Laxmi Kant | KGP Talkie |

Tokenization in LLM: 4 Quirks That Confuse Models

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER