How GPUs Work for LLMs, and Why Memory Limits LLM Speed @KGPTalkie
How GPUs Work for LLMs, and Why Memory Limits LLM Speed  @KGPTalkie
Uploaded September 2026 | Updated November 2026, 2 days ago
Your GPU is almost never short on math. It is short on data. An NVIDIA H100 can do about 989 trillion operations every second, but it can only pull 3.35 TB of data from its memory every second. In this video we walk inside the chip and see why that one gap decides how fast your AI models run.

What you will learn:

CPU vs GPU: a few big, clever cores vs an army of small ones (17,000 cores in 132 SMs on the H100)
Warps: 32 threads that run one instruction, and why branchy code wastes half of them
How a GPU hides the memory wait by switching warps, for free
The one ratio to remember: about 300 operations per byte
The memory ladder: registers, shared memory and L1, L2 cache, HBM
Why chat is memory bound: about 1 operation per byte, and the memory wall
Four tricks that beat the wall: batching, tiling, kernel fusion, quantisation
Speed ceilings for an 8B model: about 200, 400 and 800 tokens per second at 16, 8 and 4 bit
What this means on your desk: 24 GB cards, and CPU + RAM (it works, just slower)


Tokens per second = bandwidth ÷ bytes per token. So before you buy a GPU for AI, look at its memory bandwidth, not just its TFLOPS.

Watch next: Apple M5 Max vs RTX 5090, why the 5090 is 4x faster for local LLMs youtube.com/watch?v=ehb9yUCD3vc

What was the first GPU you ran a model on? Or are you running on CPU and RAM? Tell me in the comments.

Get the PDF slides: scan the QR code in the video (Instagram @kgptalkie).

Chapters:
0:00 Your GPU is short on data, not math
0:17 Fast at math, starved for data
0:38 CPU vs GPU: inside the H100
1:03 Warps: 32 threads, one instruction
1:23 Branchy code vs matrix math
1:43 Hiding the memory wait
2:26 The one ratio: 300 ops per byte
2:52 The memory ladder: registers to HBM
3:23 Why chat is memory bound
3:43 The memory wall
4:01 Four tricks: batching, tiling, fusion, quantisation
4:53 Speed ceilings: 200, 400, 800 tokens/s
5:20 Your desk: 24 GB cards, CPU and RAM
5:42 What to check before you buy a GPU

#GPU #NVIDIA #H100 #LLM #LocalLLM #AIHardware #KGPTalkie
How GPUs Work for LLMs, and Why Memory Limits LLM SpeedWhich GPU Should You Buy to Run AI Locally? #shortsGranite 4.2 8B Is Smaller - But Needs 10× the KV Cache #shortsHow to Install TensorFlow GPU on Windows 11 | GPU Setup on Windows | TensorFlow GPU SetupInstall Ubuntu 22.04 Desktop with Virtual Box in WindowsQwen 3.8 27B Speed Settings Explained: MTP, KV Cache and Flash AttentionDeepSeek Harness Desktop App: Free Setup on Windows (Ollama, No API Key)Gemma 4 31B vs Gemini 4 Argon: 5 Real Coding Tests, RTX 5090 vs the CloudQwen 3.8 27B vs Claude Opus 5.5 on RTX 5090: 4 Real Coding TestsDesign BookMyShow Seat Booking in 3D | System Design Interview Day 005Course Introduction for Fine Tuning LLM with Hugging Face Transformers for NLPGemini 3.5 Flash is INSANE! Builds Apps Before You Finish Typing 🤯
Laxmi Kant | KGP Talkie |

How GPUs Work for LLMs, and Why Memory Limits LLM Speed

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER