Subword Tokenization: Why LLMs Dont Use Words or Letters @KGPTalkie
Subword Tokenization: Why LLMs Dont Use Words or Letters  @KGPTalkie
Uploaded October 2026 | Updated November 2026, 3 days ago
Subword tokenization, an LLM interview question: why do LLMs use subword tokens? Why not whole words, or single letters? Most candidates say "because subwords are more efficient". Not wrong, but not an answer. The interviewer wants the trade-off.

- Whole words: every form needs its own entry (play, plays, played, playing), plus names, typos, hashtags and new slang. The vocabulary explodes into the millions, every entry needs its own row in the embedding table, so the model gets huge. Any word that isn't on the list becomes an unknown token the model can't read
- Single letters: a tiny vocabulary and nothing is ever unknown, but every sentence gets about 4x longer. Attention cost grows with the square of the length, the context window fills 4x faster, writing an answer takes 4x more steps, and the model has to learn spelling before meaning
- Subwords take the middle path: common words stay whole ("the"), rare words split into pieces the model already knows ("unhappiness" becomes un + happi + ness). No unknown words, a fixed vocabulary of roughly 30 thousand to a couple of hundred thousand tokens, short sentences, and pieces like "un" and "ness" are shared by lots of words
- The follow-up: the downside? Letters hide inside tokens. The model sees "strawberry" as two or three chunks, which is why LLMs struggle to count its r's. And languages the tokenizer saw less of, like Hindi, split into many more pieces

One-line answer: words blow up the vocabulary and break on unknown words, letters make sequences far too long, subwords give a fixed vocabulary, no unknown words and short sequences, at the cost of hidden spelling.

Next LLM Interview episode: a bigger vocabulary or a longer context, what's the trade-off? Comment and tell me, which answer would you have given? Follow for the full LLM Interview series.

kgptalkie.com

#shorts #LLM #Tokenizer #Tokenization #SubwordTokens #NLP #MachineLearning #AIInterview #InterviewPrep #GenerativeAI #DeepLearning #KGPTalkie
Subword Tokenization: Why LLMs Dont Use Words or LettersStreamlit Tutorial 7-1 - Working with Plotting Libraries Part 1New Ollama Desktop App: Run AI Locally with Full Privacy | Full Walkthrough & Demo🚀 New Course Launched - LangChain AI Agent Projects: Build and Deploy Real AI AgentsOrnith 1.5 Is Qwen 3.5 Underneath 🔍 #shortsFable 5.2 vs Gemini 4 Argon: Five New 3D Games, One ShotQwen 3.8 vs Muse Glimmer vs Gemma 4 Coding TestI Ran Qwen Image 2.1 on My Own GPU - Transparent Backgrounds! #shortsWhy ChatGPT Answers Word by Word | AI System Design InterviewWhy DeepSeek V4.1 Flash is so fastLangChain Tutorial for Beginners: Tools and Tool Calling ExplainedBuild Jev Like Model: Fine-Tune System 1 Decision Model
Laxmi Kant | KGP Talkie |

Subword Tokenization: Why LLMs Don't Use Words or Letters

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER