Prefill and decode want opposite things from your router @redhat
Prefill and decode want opposite things from your router  @redhat
Uploaded September 2026 | Updated November 2026, 3 hours ago
Greg Pereira, Senior Machine Learning Engineer at Red Hat, explains why one routing decision isn't enough: prefill cares deeply about landing on a pod that already holds its prefix, so it only has to compute the delta. Decode doesn't care at all, since it can pull KV from any prefiller. That's why llm-d runs separate scheduling profiles for each phase.

From vLLM Office Hours #55: Mooncake + vLLM/llm-d Deep Dive. Full session: youtu.be/RFyeBEy1AP8

Join us live every other Thursday: https://red.ht/office-hours

#vLLM #llm-d #KVCache #AIInference #RedHat #MLOps
Prefill and decode want opposite things from your routerWhy AI compliance is more than just a checkboxHow to sandbox AI agents with OpenShell | AgentOps UnlockedWho powers the Fortune 500?Scaling with OrchestratorsOpenShift simplifies cloud-native ecosystemsIs regulation accelerating tech innovation?What will you find at Red Hat Summit: Connect?Run everything youve got. Connect everywhere it counts.Run everything youve got. Connect everywhere it counts.Can we trust AI agents in the enterprise? ft. Derek Carr | Technically SpeakingWhy round-robin load balancing destroys your KV cache
Red Hat |

Prefill and decode want opposite things from your router

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER