Optimize KV cache with llm-d and vLLM @redhat
Optimize KV cache with llm-d and vLLM  @redhat
Uploaded September 2026 | Updated November 2026, 1 day ago
Discover how llm-d improves vLLM inference efficiency by preventing KV cache duplication across multi-GPU deployments. Traditional L7 load balancers use naive round-robin routing, causing up to 50% KV cache duplication and higher latency in multi-turn LLM serving. Greg Pereira, Senior Machine Learning Engineer at Red Hat, explains how llm-d introduces a prefix-based prefill and decode (PD) decider.

Learn how independent routing decisions for prefill and decode optimize prefix cache affinity and token load scoring across GPUs.

00:00 The challenge of scaling Models-as-a-Service
00:22 Why traditional L7 load balancing fails vLLM
00:45 KV cache duplication and latency issues
01:16 Selective PD disaggregation with llm-d
01:56 How the prefix-based PD decider works
02:18 Prefill vs. decode routing concerns
03:02 Calculating token load score for prefill

From vLLM Office Hours #55: Mooncake + vLLM/llm-d Deep Dive. Full session: youtu.be/RFyeBEy1AP8

Join us live every other Thursday: https://red.ht/office-hours

#vLLM #llm-d #KVCache #AIInference #RedHat #MLOps
Optimize KV cache with llm-d and vLLMStop trying to solve security mistakes with 40-page legal policies. 🛡️🤖Any workload. Any infrastructure. One platform.What is Red Hat OpenShift Virtualization?Agentic CI/CD | Ask an OpenShift Expert 180Deliver at the speed your customers expect. Anywhere.Building an agentic AI system at scaleOpenClaw and OpenShell: open source synergiesHow llm-d scales AI inference across GPU clustersHow Lightwell Network addresses open source vulnerabilitiesScale IT automation across 500 processesBreaking Down Your App Into Containers: Worth It?
Red Hat |

Optimize KV cache with llm-d and vLLM

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER