Uploaded October 2026 | Updated November 2026, 4 days ago
Same sentence, two batches, and the model gives two different answers. In an LLM Interview you get this question: "Why do we need a PAD token and an attention mask?" Most candidates say "to make short sentences the same length". The interviewer nods, writes "weak", and moves on. The real point: even after you add PAD, the model's answer must not change.
- Fix 1, why PAD: a GPU runs a batch as a rectangle, every row the same length. A four-token sentence next to a ten-token one gets six PAD tokens
- Fix 2, the attention mask: without it every token also looks at the PAD, so the output changes with the batch. The mask is 1 for a real token, 0 for PAD, and the PAD scores are set to minus infinity before the softmax, so their weight is zero
- Fix 3, the loss: PAD labels are set to -100 and PyTorch's loss ignores them. Otherwise the model learns to predict PAD
- Fix 4, which side: models like GPT add the new token at the right end, so generation uses left padding
- Fix 5, the waste: if one sentence is 500 tokens and the rest are 50, most GPU work goes into PAD. Pad each batch only to its own longest sentence and keep similar lengths together
- The trap: Llama 2's tokenizer has no PAD token. The usual trick is PAD = EOS, but if you set every label with that ID to -100 the real EOS is ignored too and the model won't learn to stop. Remove only the padding positions, or add a separate PAD token
One-line answer: PAD turns the batch into a rectangle, the mask sets its score to minus infinity so the output doesn't change, ignore PAD in the loss, pad on the left for generation, and never ignore the real EOS.
Next LLM Interview episode: why can't LLMs count the letters in a word? Comment and tell me, do you pad on the left or on the right? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #LLMInterview #AIInterview #InterviewPrep #PADToken #AttentionMask #Transformers #GenAI #MachineLearning #HuggingFace #KGPTalkie
Same sentence, two batches, and the model gives two different answers. In an LLM Interview you get this question: "Why do we need a PAD token and an attention mask?" Most candidates say "to make short sentences the same length". The interviewer nods, writes "weak", and moves on. The real point: even after you add PAD, the model's answer must not change.
- Fix 1, why PAD: a GPU runs a batch as a rectangle, every row the same length. A four-token sentence next to a ten-token one gets six PAD tokens
- Fix 2, the attention mask: without it every token also looks at the PAD, so the output changes with the batch. The mask is 1 for a real token, 0 for PAD, and the PAD scores are set to minus infinity before the softmax, so their weight is zero
- Fix 3, the loss: PAD labels are set to -100 and PyTorch's loss ignores them. Otherwise the model learns to predict PAD
- Fix 4, which side: models like GPT add the new token at the right end, so generation uses left padding
- Fix 5, the waste: if one sentence is 500 tokens and the rest are 50, most GPU work goes into PAD. Pad each batch only to its own longest sentence and keep similar lengths together
- The trap: Llama 2's tokenizer has no PAD token. The usual trick is PAD = EOS, but if you set every label with that ID to -100 the real EOS is ignored too and the model won't learn to stop. Remove only the padding positions, or add a separate PAD token
One-line answer: PAD turns the batch into a rectangle, the mask sets its score to minus infinity so the output doesn't change, ignore PAD in the loss, pad on the left for generation, and never ignore the real EOS.
Next LLM Interview episode: why can't LLMs count the letters in a word? Comment and tell me, do you pad on the left or on the right? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #LLMInterview #AIInterview #InterviewPrep #PADToken #AttentionMask #Transformers #GenAI #MachineLearning #HuggingFace #KGPTalkie










