ChatGPT System Design | API Gateway, SSE Streaming, KV Cache, Admission Control @KGPTalkie
ChatGPT System Design | API Gateway, SSE Streaming, KV Cache, Admission Control  @KGPTalkie
Uploaded November 2026 | Updated November 2026, 2 days ago
You hit stop. The GPU keeps writing anyway, for no one. In a System Design Interview you get exactly this question: "Design ChatGPT."

System Design Interview, Day 010. Most candidates say "put the model behind an API, and autoscale it". One line like that, and the round slips away. GPUs are expensive, a new GPU server takes minutes to load the model, and how many users fit depends on GPU memory, not CPU.

- Fix 1: the request path. An API gateway checks login and rate limits. The chat service loads the conversation history and trims it to the context window. A queue and a router pick a GPU replica with free capacity. The inference server generates tokens, streamed to the user over SSE, and the finished message is saved to the conversation store
- Fix 2: the GPU memory math. Say a GPU has 80 GB. The weights take 40. About 30 GB of the rest goes to the KV cache. If one long chat needs about 1 GB, that's about 30 long chats at once per GPU. Capacity comes from this memory, not from requests per second
- Fix 3: admission control and priority. When every slot is full, new requests wait in a queue instead of crashing the server, and paid users get priority. Count rate limits in tokens, not requests: one request can be 10 tokens, or 10,000
- Fix 4: prefix caching. Every request starts with the same system prompt, so build its KV cache once and reuse it. In long chats, trim the old turns or summarize them
- The follow-up: the user hits stop mid-answer, or closes the tab. In a simple design the GPU keeps generating to the end, for nobody. Catch the closed connection or the stop request at the gateway, pass the cancel to the inference server, drop the request from the batch and free its KV slot right away. Save the partial answer, so the next message has the right context

Put together, a strong answer: a clean path from the gateway to SSE, the GPU memory math, token-based rate limits and priority, prefix caching, and a cancel on stop that frees the KV slot.

Next System Design Interview episode: design Google Drive. Comment and tell me: when ChatGPT goes off track, do you hit stop?

kgptalkie.com

#shorts #SystemDesign #SystemDesignInterview #ChatGPT #LLMInference #KVCache #GPU #BackendInterview #InterviewPrep #SoftwareEngineering #DistributedSystems #KGPTalkie
ChatGPT System Design | API Gateway, SSE Streaming, KV Cache, Admission Control02 Gen AI Interview Preparation: 5 Prompt Engineering Mistakes That Kill Performance #ai #interviewWhy Does ChatGPT Type Word by Word? Streaming + GPU Serving ExplainedAI Agent Platforms Explained: Which One to Use? Copilot, Agentforce, Foundry, AgentCoreTransformers Explained in 90 Seconds: Tokenization, Embeddings, AttentionJev and Laya Explained - What is System One Model9. Install WSL 2 in Windows 11 for Python Coding with VS CodeMCP Tutorial 2 - Analyze Any Excel Sheet with Excel MCP Server and AICreating an Armored Knight in Blender | Blender MCP + Hunyuan MCP AI Workflown8n + Ollama: Build Local AI Agent with Qwen 3.8 27B (Free, Private)Byte Pair Encoding Explained | BPE Tokenizer Step by StepQwen 3.8 27B vs Muse Glimmer 30B: Is the Fast One Better?
Laxmi Kant | KGP Talkie |

ChatGPT System Design | API Gateway, SSE Streaming, KV Cache, Admission Control

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER