Uploaded November 2026 | Updated November 2026, 2 days ago
You hit stop. The GPU keeps writing anyway, for no one. In a System Design Interview you get exactly this question: "Design ChatGPT."
System Design Interview, Day 010. Most candidates say "put the model behind an API, and autoscale it". One line like that, and the round slips away. GPUs are expensive, a new GPU server takes minutes to load the model, and how many users fit depends on GPU memory, not CPU.
- Fix 1: the request path. An API gateway checks login and rate limits. The chat service loads the conversation history and trims it to the context window. A queue and a router pick a GPU replica with free capacity. The inference server generates tokens, streamed to the user over SSE, and the finished message is saved to the conversation store
- Fix 2: the GPU memory math. Say a GPU has 80 GB. The weights take 40. About 30 GB of the rest goes to the KV cache. If one long chat needs about 1 GB, that's about 30 long chats at once per GPU. Capacity comes from this memory, not from requests per second
- Fix 3: admission control and priority. When every slot is full, new requests wait in a queue instead of crashing the server, and paid users get priority. Count rate limits in tokens, not requests: one request can be 10 tokens, or 10,000
- Fix 4: prefix caching. Every request starts with the same system prompt, so build its KV cache once and reuse it. In long chats, trim the old turns or summarize them
- The follow-up: the user hits stop mid-answer, or closes the tab. In a simple design the GPU keeps generating to the end, for nobody. Catch the closed connection or the stop request at the gateway, pass the cancel to the inference server, drop the request from the batch and free its KV slot right away. Save the partial answer, so the next message has the right context
Put together, a strong answer: a clean path from the gateway to SSE, the GPU memory math, token-based rate limits and priority, prefix caching, and a cancel on stop that frees the KV slot.
Next System Design Interview episode: design Google Drive. Comment and tell me: when ChatGPT goes off track, do you hit stop?
kgptalkie.com
#shorts #SystemDesign #SystemDesignInterview #ChatGPT #LLMInference #KVCache #GPU #BackendInterview #InterviewPrep #SoftwareEngineering #DistributedSystems #KGPTalkie
You hit stop. The GPU keeps writing anyway, for no one. In a System Design Interview you get exactly this question: "Design ChatGPT."
System Design Interview, Day 010. Most candidates say "put the model behind an API, and autoscale it". One line like that, and the round slips away. GPUs are expensive, a new GPU server takes minutes to load the model, and how many users fit depends on GPU memory, not CPU.
- Fix 1: the request path. An API gateway checks login and rate limits. The chat service loads the conversation history and trims it to the context window. A queue and a router pick a GPU replica with free capacity. The inference server generates tokens, streamed to the user over SSE, and the finished message is saved to the conversation store
- Fix 2: the GPU memory math. Say a GPU has 80 GB. The weights take 40. About 30 GB of the rest goes to the KV cache. If one long chat needs about 1 GB, that's about 30 long chats at once per GPU. Capacity comes from this memory, not from requests per second
- Fix 3: admission control and priority. When every slot is full, new requests wait in a queue instead of crashing the server, and paid users get priority. Count rate limits in tokens, not requests: one request can be 10 tokens, or 10,000
- Fix 4: prefix caching. Every request starts with the same system prompt, so build its KV cache once and reuse it. In long chats, trim the old turns or summarize them
- The follow-up: the user hits stop mid-answer, or closes the tab. In a simple design the GPU keeps generating to the end, for nobody. Catch the closed connection or the stop request at the gateway, pass the cancel to the inference server, drop the request from the batch and free its KV slot right away. Save the partial answer, so the next message has the right context
Put together, a strong answer: a clean path from the gateway to SSE, the GPU memory math, token-based rate limits and priority, prefix caching, and a cancel on stop that frees the KV slot.
Next System Design Interview episode: design Google Drive. Comment and tell me: when ChatGPT goes off track, do you hit stop?
kgptalkie.com
#shorts #SystemDesign #SystemDesignInterview #ChatGPT #LLMInference #KVCache #GPU #BackendInterview #InterviewPrep #SoftwareEngineering #DistributedSystems #KGPTalkie










