Uploaded October 2026 | Updated November 2026, 3 days ago
16 GB of GPU memory, for a single user. That is what one long context can cost. Interview question: a bigger vocabulary or a longer context, what's the trade-off? Most candidates say "bigger is better, grow both". Interviewers hear that all day. Both have a cost, just in different places.
- Bigger vocabulary, the win: each token covers more text, so the same sentence needs fewer tokens. Fewer steps, a smaller bill, more text in the same context. Languages like Hindi gain the most. Llama 2 had a 32K vocabulary, Llama 3 about 128K, and Meta reported the same text taking up to 15% fewer tokens
- Bigger vocabulary, the cost: every new token adds a row to the embedding table and one more score at the output, at every step. Say each row has about 4,000 numbers: going from 32K to 128K tokens adds about 400 million numbers in the input table alone. Rare tokens show up so seldom that their rows never get learned properly
- Longer context, the win: the model sees the whole document at once
- Longer context, the cost: it comes on every request. Attention compares every token with every other token, so twice the length means four times the work. The KV cache grows with every token: in Llama 3 8B about 128 KB per token, so a 128K-token context needs about 16 GB just for the cache, for a single user
- What separates a hire from a maybe: the vocabulary cost is paid once, in parameters. The context cost is paid on every request and grows with the length. And they are linked, because a better vocabulary compresses text so more fits in the same context
- Follow-up: why not a vocabulary of a million tokens? With rows of 4,000 numbers that is 4 billion numbers per table, half of an 8 billion model, the output chooses from a million options at every step, and every extra token saves less than the one before
One-line answer: grow the vocabulary while compression still pays, and size the context to the job.
Next LLM Interview episode: asked in Hindi, why is the bill higher? Comment and tell me, for a Hindi chatbot, which one would you grow first? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #Tokenizer #ContextWindow #KVCache #NLP #MachineLearning #AIInterview #InterviewPrep #GenerativeAI #DeepLearning #KGPTalkie
16 GB of GPU memory, for a single user. That is what one long context can cost. Interview question: a bigger vocabulary or a longer context, what's the trade-off? Most candidates say "bigger is better, grow both". Interviewers hear that all day. Both have a cost, just in different places.
- Bigger vocabulary, the win: each token covers more text, so the same sentence needs fewer tokens. Fewer steps, a smaller bill, more text in the same context. Languages like Hindi gain the most. Llama 2 had a 32K vocabulary, Llama 3 about 128K, and Meta reported the same text taking up to 15% fewer tokens
- Bigger vocabulary, the cost: every new token adds a row to the embedding table and one more score at the output, at every step. Say each row has about 4,000 numbers: going from 32K to 128K tokens adds about 400 million numbers in the input table alone. Rare tokens show up so seldom that their rows never get learned properly
- Longer context, the win: the model sees the whole document at once
- Longer context, the cost: it comes on every request. Attention compares every token with every other token, so twice the length means four times the work. The KV cache grows with every token: in Llama 3 8B about 128 KB per token, so a 128K-token context needs about 16 GB just for the cache, for a single user
- What separates a hire from a maybe: the vocabulary cost is paid once, in parameters. The context cost is paid on every request and grows with the length. And they are linked, because a better vocabulary compresses text so more fits in the same context
- Follow-up: why not a vocabulary of a million tokens? With rows of 4,000 numbers that is 4 billion numbers per table, half of an 8 billion model, the output chooses from a million options at every step, and every extra token saves less than the one before
One-line answer: grow the vocabulary while compression still pays, and size the context to the job.
Next LLM Interview episode: asked in Hindi, why is the bill higher? Comment and tell me, for a Hindi chatbot, which one would you grow first? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #Tokenizer #ContextWindow #KVCache #NLP #MachineLearning #AIInterview #InterviewPrep #GenerativeAI #DeepLearning #KGPTalkie










