Attention Mask in Transformer: Why Do We Need a PAD Token? @KGPTalkie
Attention Mask in Transformer: Why Do We Need a PAD Token?  @KGPTalkie
Uploaded October 2026 | Updated November 2026, 4 days ago
Same sentence, two batches, and the model gives two different answers. In an LLM Interview you get this question: "Why do we need a PAD token and an attention mask?" Most candidates say "to make short sentences the same length". The interviewer nods, writes "weak", and moves on. The real point: even after you add PAD, the model's answer must not change.

- Fix 1, why PAD: a GPU runs a batch as a rectangle, every row the same length. A four-token sentence next to a ten-token one gets six PAD tokens
- Fix 2, the attention mask: without it every token also looks at the PAD, so the output changes with the batch. The mask is 1 for a real token, 0 for PAD, and the PAD scores are set to minus infinity before the softmax, so their weight is zero
- Fix 3, the loss: PAD labels are set to -100 and PyTorch's loss ignores them. Otherwise the model learns to predict PAD
- Fix 4, which side: models like GPT add the new token at the right end, so generation uses left padding
- Fix 5, the waste: if one sentence is 500 tokens and the rest are 50, most GPU work goes into PAD. Pad each batch only to its own longest sentence and keep similar lengths together
- The trap: Llama 2's tokenizer has no PAD token. The usual trick is PAD = EOS, but if you set every label with that ID to -100 the real EOS is ignored too and the model won't learn to stop. Remove only the padding positions, or add a separate PAD token

One-line answer: PAD turns the batch into a rectangle, the mask sets its score to minus infinity so the output doesn't change, ignore PAD in the loss, pad on the left for generation, and never ignore the real EOS.

Next LLM Interview episode: why can't LLMs count the letters in a word? Comment and tell me, do you pad on the left or on the right? Follow for the full LLM Interview series.

kgptalkie.com

#shorts #LLM #LLMInterview #AIInterview #InterviewPrep #PADToken #AttentionMask #Transformers #GenAI #MachineLearning #HuggingFace #KGPTalkie
Attention Mask in Transformer: Why Do We Need a PAD Token?Qwen 3.8 27B vs GPT-6.1 Sol: RTX 5090 vs the Cloud, 4 Real Coding TestsGen AI Interview #11: Greedy Decoding vs Beam Search - How LLMs Choose Their Next Word Asked in METAQwen 3.8 27B on RTX 5090 vs Claude Opus 5.5: 4 Real TestsDeepSeek Just Gave Away V4.1 Flash for FREE 🤯 #shortsHow Is a Tokenizer Trained? BPE in 5 Steps | LLM ConceptsBenQ 240Q Programming Monitor Box Opening | My Work Station SetupWhat Is a Token in AI? What an LLM Really ReadsNLP Projects 1 - Build Spam Message Classification Application with Streamlit | Build Streamlit AppEmbeddings in LLM: How Token IDs Become VectorsJioHotstar System Design: Why Autoscaling Fails at IPL Toss13 Gen AI Interview Preparation: What is MLM Maked Language Modelling
Laxmi Kant | KGP Talkie |

Attention Mask in Transformer: Why Do We Need a PAD Token?

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER