Uploaded October 2026 | Updated November 2026, 3 days ago
Token IDs are just labels, like roll numbers. So how does a model get meaning out of them? Through embeddings: a learned row of numbers for every token.
- The problem: the tokenizer turns "cat" into an ID such as 500. That number is a label with no meaning, ID 500 and ID 501 have nothing to do with each other, and feeding the raw number would make the model think 501 is a little bigger than 500
- The embedding table: one row per token, and each row is a long list of numbers. GPT-2 small has more than fifty thousand rows, each holding 768 numbers
- Step one, lookup: ID 500 comes in and you pick up row 500 of the table. No calculation. A six-token sentence becomes six vectors of 768 numbers each
- Step two, the numbers are learned: every row starts random. In training the model guesses the next token, gets it wrong, and backpropagation nudges every row that was used
- Step three, similar meanings move closer: tokens that appear in similar contexts end up with vectors that sit close together. We measure closeness with cosine similarity, the angle between two vectors. With just two numbers per vector: cat (9, 1) and dog (8, 2) point the same way, car (1, 9) points somewhere else
- Step four, directions carry meaning: the famous word2vec example, king minus man plus woman lands near queen
- The catch: the input embedding has no context. "Bank" in "river bank" and in "bank account" gets exactly the same vector from the table; the attention layers mix the context in later
- The size: in GPT-2 small this table alone holds nearly forty million parameters
- Illustrative on screen: the numbers inside the table rows and vector bars, the extra token IDs (17, 3301, 501, 31, 263, 91, 902), the 2D positions of king, man, woman, queen, chai, coffee and cricket, the background dots, the cosine values 0.99 and 0.22, the "tree" guess and "mat" answer, the river / bank / account vectors and the LLM Interview card
Coming up next in LLM Interview: how does the lookup really work inside, and how do those rows get updated in training? If a vector had just two numbers, where would you put chai: next to coffee, or next to cricket? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Embeddings #WordEmbeddings #Word2Vec #TokenIDs #Transformers #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie
Token IDs are just labels, like roll numbers. So how does a model get meaning out of them? Through embeddings: a learned row of numbers for every token.
- The problem: the tokenizer turns "cat" into an ID such as 500. That number is a label with no meaning, ID 500 and ID 501 have nothing to do with each other, and feeding the raw number would make the model think 501 is a little bigger than 500
- The embedding table: one row per token, and each row is a long list of numbers. GPT-2 small has more than fifty thousand rows, each holding 768 numbers
- Step one, lookup: ID 500 comes in and you pick up row 500 of the table. No calculation. A six-token sentence becomes six vectors of 768 numbers each
- Step two, the numbers are learned: every row starts random. In training the model guesses the next token, gets it wrong, and backpropagation nudges every row that was used
- Step three, similar meanings move closer: tokens that appear in similar contexts end up with vectors that sit close together. We measure closeness with cosine similarity, the angle between two vectors. With just two numbers per vector: cat (9, 1) and dog (8, 2) point the same way, car (1, 9) points somewhere else
- Step four, directions carry meaning: the famous word2vec example, king minus man plus woman lands near queen
- The catch: the input embedding has no context. "Bank" in "river bank" and in "bank account" gets exactly the same vector from the table; the attention layers mix the context in later
- The size: in GPT-2 small this table alone holds nearly forty million parameters
- Illustrative on screen: the numbers inside the table rows and vector bars, the extra token IDs (17, 3301, 501, 31, 263, 91, 902), the 2D positions of king, man, woman, queen, chai, coffee and cricket, the background dots, the cosine values 0.99 and 0.22, the "tree" guess and "mat" answer, the river / bank / account vectors and the LLM Interview card
Coming up next in LLM Interview: how does the lookup really work inside, and how do those rows get updated in training? If a vector had just two numbers, where would you put chai: next to coffee, or next to cricket? Comment and tell me. Follow for the LLM Concepts series.
kgptalkie.com
#shorts #LLM #Embeddings #WordEmbeddings #Word2Vec #TokenIDs #Transformers #GenerativeAI #MachineLearning #AIExplained #LLMConcepts #KGPTalkie










