Uploaded August 2026 | Updated September 2026, 3 weeks ago
Agentic sessions in Red Hat's traces run from a few turns to 3,000, cache hit rates routinely clear 90%, and input to output token ratios often pass 100 to 1. A public inference benchmark shows none of that, because it reports steady state numbers from one sanitized run. Yuchen Fama and Ashish Kamra spend the talk on the two levers that matter once the client rather than the server controls the cache lifecycle, and a live demo makes the first one concrete. An opening request takes about 3 seconds, the next turn reuses the cache on the same pod and takes about 1, and a fresh system prompt lands on a different pod and pays the full 3 again. With a 10x gap between cached and uncached token costs, routing is the cheaper lever to reach for before adding GPUs.
The second lever splits compute bound prefill from memory bound decode, so a long incoming prompt cannot stall token generation midstream. Across 16 H100s serving gpt-oss, P99 inter token latency falls from roughly 900 milliseconds to about 100, and the curve gets visibly smoother. The useful part is where they draw the boundary. Disaggregation wins in the middle concurrency band, roughly ties at both ends, and needs an RDMA or RoCE fabric to move cache between workers at all. Without one, stay aggregated. The closing case study runs GLM 5.2 on the H200s customers actually have instead of B200s, at three prefill workers to one decode, for 4x faster time to first token and 60% more requests.
Speaker info:
- linkedin.com/in/yuchen-fama
- linkedin.com/in/ashishkamra
- github.com/llm-d/llm-d
Timestamps:
0:00 - What public inference benchmarks leave out
1:28 - Red Hat's inference stack, and the agenda
3:29 - Agentic traces: 3,000 turns, 90% cache hits, 100 to 1 ratios
5:12 - Volatile cache, and the 10x cached token gap
6:28 - How llm-d routes: endpoint picker, offload tiers, eviction
7:46 - Demo: cache hits, pod affinity, 3 seconds versus 1
9:28 - What llm-d is, and why prefill and decode interfere
12:03 - How disaggregation works in practice
13:08 - P99 inter token latency: 900ms versus 100ms
14:14 - Where PD shines across the concurrency curve
15:45 - When to use PD, and when to stay aggregated
18:10 - GLM 5.2 on H200s: 4x faster TTFT
Agentic sessions in Red Hat's traces run from a few turns to 3,000, cache hit rates routinely clear 90%, and input to output token ratios often pass 100 to 1. A public inference benchmark shows none of that, because it reports steady state numbers from one sanitized run. Yuchen Fama and Ashish Kamra spend the talk on the two levers that matter once the client rather than the server controls the cache lifecycle, and a live demo makes the first one concrete. An opening request takes about 3 seconds, the next turn reuses the cache on the same pod and takes about 1, and a fresh system prompt lands on a different pod and pays the full 3 again. With a 10x gap between cached and uncached token costs, routing is the cheaper lever to reach for before adding GPUs.
The second lever splits compute bound prefill from memory bound decode, so a long incoming prompt cannot stall token generation midstream. Across 16 H100s serving gpt-oss, P99 inter token latency falls from roughly 900 milliseconds to about 100, and the curve gets visibly smoother. The useful part is where they draw the boundary. Disaggregation wins in the middle concurrency band, roughly ties at both ends, and needs an RDMA or RoCE fabric to move cache between workers at all. Without one, stay aggregated. The closing case study runs GLM 5.2 on the H200s customers actually have instead of B200s, at three prefill workers to one decode, for 4x faster time to first token and 60% more requests.
Speaker info:
- linkedin.com/in/yuchen-fama
- linkedin.com/in/ashishkamra
- github.com/llm-d/llm-d
Timestamps:
0:00 - What public inference benchmarks leave out
1:28 - Red Hat's inference stack, and the agenda
3:29 - Agentic traces: 3,000 turns, 90% cache hits, 100 to 1 ratios
5:12 - Volatile cache, and the 10x cached token gap
6:28 - How llm-d routes: endpoint picker, offload tiers, eviction
7:46 - Demo: cache hits, pod affinity, 3 seconds versus 1
9:28 - What llm-d is, and why prefill and decode interfere
12:03 - How disaggregation works in practice
13:08 - P99 inter token latency: 900ms versus 100ms
14:14 - Where PD shines across the concurrency curve
15:45 - When to use PD, and when to stay aggregated
18:10 - GLM 5.2 on H200s: 4x faster TTFT
