Uploaded October 2025 | Updated September 2026, 2 weeks ago
Resources:
- Understanding and Coding the KV Cache in LLMs from Scratch: magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms
- The Big Architecture Comparison: magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison
- Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers
- Reasoning From Scratch book: https://mng.bz/Nwr7
Description:
Learn the core components of modern transformer-based large language models (LLMs) and the practical techniques that make inference faster and cheaper.
We walk through Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Sliding Window Attention (SWA), show where Mixture of Experts (MoE) fits into today’s architectures, and finish with a look at promising alternatives and hybrid models beyond standard transformers.
Chapters:
00:00 Intro
01:13 Main theme: bigger models and cheaper inference
02:13 Grouped-Query Attention (GQA)
05:44 Multi-Head Latent Attention (MLA)
09:51 Sliding Window Attention (SWA)
13:57 Mixture of Experts
17:01 LLM and transformer alternatives
#LLM #Transformers #DeepLearning #MachineLearning #Inference #MoE #GQA #MLA #SWA #KVCache
Resources:
- Understanding and Coding the KV Cache in LLMs from Scratch: magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms
- The Big Architecture Comparison: magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison
- Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers
- Reasoning From Scratch book: https://mng.bz/Nwr7
Description:
Learn the core components of modern transformer-based large language models (LLMs) and the practical techniques that make inference faster and cheaper.
We walk through Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Sliding Window Attention (SWA), show where Mixture of Experts (MoE) fits into today’s architectures, and finish with a look at promising alternatives and hybrid models beyond standard transformers.
Chapters:
00:00 Intro
01:13 Main theme: bigger models and cheaper inference
02:13 Grouped-Query Attention (GQA)
05:44 Multi-Head Latent Attention (MLA)
09:51 Sliding Window Attention (SWA)
13:57 Mixture of Experts
17:01 LLM and transformer alternatives
#LLM #Transformers #DeepLearning #MachineLearning #Inference #MoE #GQA #MLA #SWA #KVCache
