Uploaded October 2026 | Updated November 2026, 1 day ago
Interview question: BPE, WordPiece and SentencePiece. What's the difference, and which one does your model use? Most candidates say "all three split words into pieces". True, but that gets no marks. The interviewer wants the one rule that sets each one apart.
- BPE: count every neighbouring pair, merge the most frequent one, repeat. GPT models use byte-level BPE, which starts from 256 bytes, so it never meets a character it can't write. GPT, RoBERTa and Llama 3 run on BPE
- WordPiece: picks the merge that helps predict the training text best, the pair that shows up together far more often than its parts do on their own. Spot it by ##: "playing" becomes play ##ing, and ## means the piece continues the word. That's the BERT family
- SentencePiece: changes the input, not the rule. No splitting on spaces first; it reads one raw stream and turns every space into a small mark, so "New York" becomes ▁New ▁York. That is why it works on Japanese and Chinese, and why decoding gives back the exact text. Inside it runs Unigram or BPE. T5 and Llama 2 use it
- The follow-up trap: can you use BERT's tokenizer with Llama? No. A token ID is just a row number in the model's embedding table. In another model the same ID means a different piece. No error, the model just reads nonsense
One-line answer: BPE merges the most frequent pair, WordPiece merges by likelihood and marks inner pieces with ##, SentencePiece treats the space as a character, and the tokenizer must always match the model.
Next LLM Interview episode: why subword tokens, and not whole words or single letters? Comment and tell me, which tokenizer does your favourite model use? Follow for the full LLM Interview series.
kgptalkie.com
Learn more with my Udemy courses:
- Advanced RAG: Build & Deploy Production GenAI Apps: kgptalkie.com/advanced-rag
- Build and Deploy AI Agents with Gemini and Langchain: kgptalkie.com/ai-agent-projects
- Agentic AI: Private Agentic RAG with LangGraph v1 & Ollama: kgptalkie.com/agentic-rag
- Deep Agent: Multi Agent RAG with Gemini and Langchain: kgptalkie.com/deep-agent
#shorts #LLM #Tokenizer #BPE #WordPiece #SentencePiece #NLP #MachineLearning #AIInterview #InterviewPrep #GenerativeAI #KGPTalkie
Interview question: BPE, WordPiece and SentencePiece. What's the difference, and which one does your model use? Most candidates say "all three split words into pieces". True, but that gets no marks. The interviewer wants the one rule that sets each one apart.
- BPE: count every neighbouring pair, merge the most frequent one, repeat. GPT models use byte-level BPE, which starts from 256 bytes, so it never meets a character it can't write. GPT, RoBERTa and Llama 3 run on BPE
- WordPiece: picks the merge that helps predict the training text best, the pair that shows up together far more often than its parts do on their own. Spot it by ##: "playing" becomes play ##ing, and ## means the piece continues the word. That's the BERT family
- SentencePiece: changes the input, not the rule. No splitting on spaces first; it reads one raw stream and turns every space into a small mark, so "New York" becomes ▁New ▁York. That is why it works on Japanese and Chinese, and why decoding gives back the exact text. Inside it runs Unigram or BPE. T5 and Llama 2 use it
- The follow-up trap: can you use BERT's tokenizer with Llama? No. A token ID is just a row number in the model's embedding table. In another model the same ID means a different piece. No error, the model just reads nonsense
One-line answer: BPE merges the most frequent pair, WordPiece merges by likelihood and marks inner pieces with ##, SentencePiece treats the space as a character, and the tokenizer must always match the model.
Next LLM Interview episode: why subword tokens, and not whole words or single letters? Comment and tell me, which tokenizer does your favourite model use? Follow for the full LLM Interview series.
kgptalkie.com
Learn more with my Udemy courses:
- Advanced RAG: Build & Deploy Production GenAI Apps: kgptalkie.com/advanced-rag
- Build and Deploy AI Agents with Gemini and Langchain: kgptalkie.com/ai-agent-projects
- Agentic AI: Private Agentic RAG with LangGraph v1 & Ollama: kgptalkie.com/agentic-rag
- Deep Agent: Multi Agent RAG with Gemini and Langchain: kgptalkie.com/deep-agent
#shorts #LLM #Tokenizer #BPE #WordPiece #SentencePiece #NLP #MachineLearning #AIInterview #InterviewPrep #GenerativeAI #KGPTalkie










