Production-Grade Distributed Inference with llm-d | Red Hat | Ray Summit 2026 @anyscale
Production-Grade Distributed Inference with llm-d | Red Hat | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 13 hours ago
llm-d brings production-grade orchestration and distributed inference optimizations to vLLM deployments on Kubernetes.

At Ray Summit 2026, Greg Pereira and Michael Goin from Red Hat explore how intelligent prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV cache offloading, and traffic- and hardware-aware scheduling improve throughput and reliability across multi-node serving clusters, and how llm-d integrates vLLM with the Kubernetes Gateway API and operational tooling to create a modular serving stack.

You'll leave knowing how to deploy, scale, and tune llm-d for real-world production traffic.

This session was part of the first vLLM Conference at Ray Summit, hosted by Inferact.

Liked this video? Check out other Ray Summit vLLM session recordings: youtube.com/playlist?list=PLXMguE8Nc9o4

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Production-Grade Distributed Inference with llm-d | Red Hat | Ray Summit 2026Scaling RL with Ray, Kubernetes, and TPUs | Google | Ray Summit 2026How to Build a Fully Autonomous Excavator | Vincent Gonguet (Bedrock Robotics) | Ray Summit 2026Performant Training for All Workstreams | Latitude AI | Ray Summit 2026How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025MLOps with Ray on Anyscale | Ray Summit 2025Distributed Embeddings at Scale: Processing 10M+ Rows/ Day with Ray, GPUs & Qdrant | Ray Summit 2025Scaling Physical AI & Robotics Systems with Ray | Anyscale | Ray Summit 2026Serving vLLM on Intel GPUs, CPUs, and Gaudi | Intel | Ray Summit 2026Elastic Expert Parallelism for vLLM | Ray Summit 2025Post-training Nemotron with Ray | Bryan Catanzaro (Nvidia) | Ray Summit 2026Apple’s Approach to Scalable Machine Learning Infrastructure on Ray | Ray Summit 2025
Anyscale |

Production-Grade Distributed Inference with llm-d | Red Hat | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER