Uploaded September 2026 | Updated November 2026, 5 hours ago
A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text.
An excerpt from my reasoning model series: youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1
A tokenizer splits text into tokens and maps them to numeric IDs. I walk through this process with the Qwen3 tokenizer, then show how decoding reconstructs the original text.
An excerpt from my reasoning model series: youtube.com/watch?v=Kh9mqTzjuEQ&list=PLfrHAQCRxtUs&index=1










