Why Does ChatGPT Type Word by Word? Streaming + GPU Serving Explained @KGPTalkie
Why Does ChatGPT Type Word by Word? Streaming + GPU Serving Explained  @KGPTalkie
Uploaded November 2026 | Updated November 2026, 2 days ago
Six seconds staring at a blank screen, or words in under a second?

Why does ChatGPT look like it's typing its answer word by word? An LLM produces one token at a time. Say an answer is 300 tokens and the model writes 50 tokens per second: the full answer takes 6 seconds. Wait for all of it, and the user stares at a blank screen.

Step 1, streaming: send every token the moment it's made. The usual way is SSE, server-sent events: one HTTP response stays open and the server pushes each new token as a small event. The first words show up in under a second.

Step 2, batching: for every token the GPU reads all the model's weights from memory, so for one user the compute cores mostly sit idle. Generate the next token for, say, 32 users in one pass: weights read once, 32 tokens instead of one. Each user is a little slower, the GPU's total output is many times higher.

Step 3, continuous batching: a static batch waits for its longest answer (20 tokens vs 500, the short one's slot sits empty). Continuous batching lets finished requests leave and new ones join after every step.

Step 4, KV cache memory: every request keeps the keys and values of every token in GPU memory, so memory, not compute, decides how many users run at once. Reserving the maximum length up front wastes most of it; vLLM's PagedAttention stores the cache in small fixed-size blocks, like an operating system's memory pages, handed out as needed. Far more requests fit on the same GPU.

The catch: streaming doesn't finish sooner, 6 seconds is still 6 seconds, it just feels shorter. A proxy or load balancer that buffers the response breaks streaming. And every stream is a long-lived connection, so the gateway holds thousands of them.

Next: in today's System Design Interview reel, design ChatGPT. Comment and tell me: answers streamed, or all at once? Follow for the full series.

kgptalkie.com

#shorts #SystemDesign #LLM #ChatGPT #GPU #Streaming #KVCache #vLLM #Backend #DistributedSystems #SystemDesignConcepts #KGPTalkie
Why Does ChatGPT Type Word by Word? Streaming + GPU Serving ExplainedAI Agent Platforms Explained: Which One to Use? Copilot, Agentforce, Foundry, AgentCoreTransformers Explained in 90 Seconds: Tokenization, Embeddings, AttentionJev and Laya Explained - What is System One Model9. Install WSL 2 in Windows 11 for Python Coding with VS CodeMCP Tutorial 2 - Analyze Any Excel Sheet with Excel MCP Server and AICreating an Armored Knight in Blender | Blender MCP + Hunyuan MCP AI Workflown8n + Ollama: Build Local AI Agent with Qwen 3.8 27B (Free, Private)Byte Pair Encoding Explained | BPE Tokenizer Step by StepQwen 3.8 27B vs Muse Glimmer 30B: Is the Fast One Better?LangChain Tutorial for Beginners: Build an AI Agent with MemorySwiggy Zomato System Design: 10x Orders at 8 PM
Laxmi Kant | KGP Talkie |

Why Does ChatGPT Type Word by Word? Streaming + GPU Serving Explained

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER