Uploaded November 2026 | Updated November 2026, 3 days ago
About a third of GPT-2 small is one single table. In an LLM Interview you get exactly this question: "How do token IDs become embeddings?" Most candidates say "an embedding layer converts them into vectors", and that is where the round slips away. The interviewer wants to know what that layer actually does.
- Point 1, a lookup: a matrix with one row per token and d model numbers per row. GPT-2 small has 50,257 rows and 768 columns. The token ID is simply the row number. A batch of 2 sentences with 10 tokens each comes out as a 2 x 10 x 768 tensor
- Point 2, one-hot: turn the token into a one-hot vector, thousands of zeros and a single one at its ID. Multiply it by the matrix and exactly that one row comes out. So an embedding layer is a linear layer with no bias, and the lookup just skips the multiplications by zero
- Point 3, why not the ID itself: a number has order and distance. The model would treat IDs 500 and 501 as neighbours, when the two tokens have nothing to do with each other
- Point 4, learning the rows: they start random, and the gradient flows back only to the rows whose tokens were in the batch. Rare tokens keep rows close to random, the usual explanation for glitch tokens like SolidGoldMagikarp
- Follow-up, how many parameters: rows times columns. GPT-2 small is nearly 40M, about a third of the model. Llama 3 8B, about 128K x 4,096, is about 525M in the embeddings alone
- Output side: a matrix of the same shape turns vectors back into scores over the vocabulary. GPT-2 shares the two matrices, Llama 3 8B doesn't. That is weight tying
One-line answer: the ID is a row number, the lookup is a one-hot multiplication, only the rows in the batch get updated, and the table is big.
Next LLM Interview episode: millions of people ask the same question, so why pay the LLM every time? Comment and tell me, which part was new to you, the one-hot trick or the glitch tokens? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #LLMInterview #AIInterview #InterviewPrep #Embeddings #Tokenization #GPT2 #Transformers #GenAI #MachineLearning #KGPTalkie
About a third of GPT-2 small is one single table. In an LLM Interview you get exactly this question: "How do token IDs become embeddings?" Most candidates say "an embedding layer converts them into vectors", and that is where the round slips away. The interviewer wants to know what that layer actually does.
- Point 1, a lookup: a matrix with one row per token and d model numbers per row. GPT-2 small has 50,257 rows and 768 columns. The token ID is simply the row number. A batch of 2 sentences with 10 tokens each comes out as a 2 x 10 x 768 tensor
- Point 2, one-hot: turn the token into a one-hot vector, thousands of zeros and a single one at its ID. Multiply it by the matrix and exactly that one row comes out. So an embedding layer is a linear layer with no bias, and the lookup just skips the multiplications by zero
- Point 3, why not the ID itself: a number has order and distance. The model would treat IDs 500 and 501 as neighbours, when the two tokens have nothing to do with each other
- Point 4, learning the rows: they start random, and the gradient flows back only to the rows whose tokens were in the batch. Rare tokens keep rows close to random, the usual explanation for glitch tokens like SolidGoldMagikarp
- Follow-up, how many parameters: rows times columns. GPT-2 small is nearly 40M, about a third of the model. Llama 3 8B, about 128K x 4,096, is about 525M in the embeddings alone
- Output side: a matrix of the same shape turns vectors back into scores over the vocabulary. GPT-2 shares the two matrices, Llama 3 8B doesn't. That is weight tying
One-line answer: the ID is a row number, the lookup is a one-hot multiplication, only the rows in the batch get updated, and the table is big.
Next LLM Interview episode: millions of people ask the same question, so why pay the LLM every time? Comment and tell me, which part was new to you, the one-hot trick or the glitch tokens? Follow for the full LLM Interview series.
kgptalkie.com
#shorts #LLM #LLMInterview #AIInterview #InterviewPrep #Embeddings #Tokenization #GPT2 #Transformers #GenAI #MachineLearning #KGPTalkie










