Uploaded September 2026 | Updated November 2026, 13 hours ago
llm-d brings production-grade orchestration and distributed inference optimizations to vLLM deployments on Kubernetes.
At Ray Summit 2026, Greg Pereira and Michael Goin from Red Hat explore how intelligent prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV cache offloading, and traffic- and hardware-aware scheduling improve throughput and reliability across multi-node serving clusters, and how llm-d integrates vLLM with the Kubernetes Gateway API and operational tooling to create a modular serving stack.
You'll leave knowing how to deploy, scale, and tune llm-d for real-world production traffic.
This session was part of the first vLLM Conference at Ray Summit, hosted by Inferact.
Liked this video? Check out other Ray Summit vLLM session recordings: youtube.com/playlist?list=PLXMguE8Nc9o4
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
llm-d brings production-grade orchestration and distributed inference optimizations to vLLM deployments on Kubernetes.
At Ray Summit 2026, Greg Pereira and Michael Goin from Red Hat explore how intelligent prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV cache offloading, and traffic- and hardware-aware scheduling improve throughput and reliability across multi-node serving clusters, and how llm-d integrates vLLM with the Kubernetes Gateway API and operational tooling to create a modular serving stack.
You'll leave knowing how to deploy, scale, and tune llm-d for real-world production traffic.
This session was part of the first vLLM Conference at Ray Summit, hosted by Inferact.
Liked this video? Check out other Ray Summit vLLM session recordings: youtube.com/playlist?list=PLXMguE8Nc9o4
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










