Uploaded January 2026 | Updated November 2026, 6 days ago
This video dives deep into Token Routing, the core algorithm of Mixture of Experts (MoE) models.
Slides: huggingface.co/ariG23498/moe-routing-algorithm/blob/main/MoE_routing_algorithm.pdf
Colab Notebook: huggingface.co/ariG23498/moe-routing-algorithm/blob/main/routing_mechanism.ipynb
Chapter Timestamp:
Introduction: 00:00
Laying the Foundation for Mixture of Experts (MoE): 00:09
Focus on Token Routing: 00:50
What is a Mixture of Experts Layer?: 02:36
Problem Statement and Configurations: 04:48
Compute Router Logits: 08:31
Sparsity and Selecting Top K Experts: 10:54
Normalizing Logits to Router Probabilities: 12:43
Slot Selection: 14:39
Dropping Oversubscribed Tokens: 16:51
Updated Normalized Token Weights: 20:36
Updated Slot Selection and Token Slots: 21:34
Final Weight Matrix Construction: 24:35
Conclusion: 32:41
Correction:
As @denisflavius5365 rightly suggested the router matrix in the slides should be of the shape 3x4 and not 4x4.
This video dives deep into Token Routing, the core algorithm of Mixture of Experts (MoE) models.
Slides: huggingface.co/ariG23498/moe-routing-algorithm/blob/main/MoE_routing_algorithm.pdf
Colab Notebook: huggingface.co/ariG23498/moe-routing-algorithm/blob/main/routing_mechanism.ipynb
Chapter Timestamp:
Introduction: 00:00
Laying the Foundation for Mixture of Experts (MoE): 00:09
Focus on Token Routing: 00:50
What is a Mixture of Experts Layer?: 02:36
Problem Statement and Configurations: 04:48
Compute Router Logits: 08:31
Sparsity and Selecting Top K Experts: 10:54
Normalizing Logits to Router Probabilities: 12:43
Slot Selection: 14:39
Dropping Oversubscribed Tokens: 16:51
Updated Normalized Token Weights: 20:36
Updated Slot Selection and Token Slots: 21:34
Final Weight Matrix Construction: 24:35
Conclusion: 32:41
Correction:
As @denisflavius5365 rightly suggested the router matrix in the slides should be of the shape 3x4 and not 4x4.










