Uploaded September 2026 | Updated November 2026, 2 days ago
Your GPU is almost never short on math. It is short on data. An NVIDIA H100 can do about 989 trillion operations every second, but it can only pull 3.35 TB of data from its memory every second. In this video we walk inside the chip and see why that one gap decides how fast your AI models run.
What you will learn:
CPU vs GPU: a few big, clever cores vs an army of small ones (17,000 cores in 132 SMs on the H100)
Warps: 32 threads that run one instruction, and why branchy code wastes half of them
How a GPU hides the memory wait by switching warps, for free
The one ratio to remember: about 300 operations per byte
The memory ladder: registers, shared memory and L1, L2 cache, HBM
Why chat is memory bound: about 1 operation per byte, and the memory wall
Four tricks that beat the wall: batching, tiling, kernel fusion, quantisation
Speed ceilings for an 8B model: about 200, 400 and 800 tokens per second at 16, 8 and 4 bit
What this means on your desk: 24 GB cards, and CPU + RAM (it works, just slower)
Tokens per second = bandwidth ÷ bytes per token. So before you buy a GPU for AI, look at its memory bandwidth, not just its TFLOPS.
Watch next: Apple M5 Max vs RTX 5090, why the 5090 is 4x faster for local LLMs youtube.com/watch?v=ehb9yUCD3vc
What was the first GPU you ran a model on? Or are you running on CPU and RAM? Tell me in the comments.
Get the PDF slides: scan the QR code in the video (Instagram @kgptalkie).
Chapters:
0:00 Your GPU is short on data, not math
0:17 Fast at math, starved for data
0:38 CPU vs GPU: inside the H100
1:03 Warps: 32 threads, one instruction
1:23 Branchy code vs matrix math
1:43 Hiding the memory wait
2:26 The one ratio: 300 ops per byte
2:52 The memory ladder: registers to HBM
3:23 Why chat is memory bound
3:43 The memory wall
4:01 Four tricks: batching, tiling, fusion, quantisation
4:53 Speed ceilings: 200, 400, 800 tokens/s
5:20 Your desk: 24 GB cards, CPU and RAM
5:42 What to check before you buy a GPU
#GPU #NVIDIA #H100 #LLM #LocalLLM #AIHardware #KGPTalkie
Your GPU is almost never short on math. It is short on data. An NVIDIA H100 can do about 989 trillion operations every second, but it can only pull 3.35 TB of data from its memory every second. In this video we walk inside the chip and see why that one gap decides how fast your AI models run.
What you will learn:
CPU vs GPU: a few big, clever cores vs an army of small ones (17,000 cores in 132 SMs on the H100)
Warps: 32 threads that run one instruction, and why branchy code wastes half of them
How a GPU hides the memory wait by switching warps, for free
The one ratio to remember: about 300 operations per byte
The memory ladder: registers, shared memory and L1, L2 cache, HBM
Why chat is memory bound: about 1 operation per byte, and the memory wall
Four tricks that beat the wall: batching, tiling, kernel fusion, quantisation
Speed ceilings for an 8B model: about 200, 400 and 800 tokens per second at 16, 8 and 4 bit
What this means on your desk: 24 GB cards, and CPU + RAM (it works, just slower)
Tokens per second = bandwidth ÷ bytes per token. So before you buy a GPU for AI, look at its memory bandwidth, not just its TFLOPS.
Watch next: Apple M5 Max vs RTX 5090, why the 5090 is 4x faster for local LLMs youtube.com/watch?v=ehb9yUCD3vc
What was the first GPU you ran a model on? Or are you running on CPU and RAM? Tell me in the comments.
Get the PDF slides: scan the QR code in the video (Instagram @kgptalkie).
Chapters:
0:00 Your GPU is short on data, not math
0:17 Fast at math, starved for data
0:38 CPU vs GPU: inside the H100
1:03 Warps: 32 threads, one instruction
1:23 Branchy code vs matrix math
1:43 Hiding the memory wait
2:26 The one ratio: 300 ops per byte
2:52 The memory ladder: registers to HBM
3:23 Why chat is memory bound
3:43 The memory wall
4:01 Four tricks: batching, tiling, fusion, quantisation
4:53 Speed ceilings: 200, 400, 800 tokens/s
5:20 Your desk: 24 GB cards, CPU and RAM
5:42 What to check before you buy a GPU
#GPU #NVIDIA #H100 #LLM #LocalLLM #AIHardware #KGPTalkie










