How Red Hat Scales Large-Scale Serving with vLLM | Ray Summit 2025 @anyscale
How Red Hat Scales Large-Scale Serving with vLLM | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 22 hours ago
At Ray Summit 2025, Robert Shaw from Red Hat shares how vLLM is evolving to support p/d disaggregation and wide expert parallelism, two critical techniques for serving frontier-scale models like DeepSeek-R1 across large GPU clusters.

He begins by breaking down how prefill/decode (p/d) disaggregation enables more efficient resource utilization by separating the heavy, bursty prefill phase from the latency-sensitive decode phase—allowing clusters to scale each stage independently for maximum throughput and cost efficiency.

Robert then dives into the implementation of wide expert parallelism (EP) in vLLM, explaining how it allows MoE-based models to leverage large numbers of experts across multi-node environments. He details the orchestration, scheduling, and memory-management challenges of EP at cluster scale, and the design decisions that make these deployments practical.

Finally, he explores the tradeoffs and system-level considerations that arise when serving massive models such as DeepSeek-R1—including GPU topology, communication overhead, batching behavior, and cluster elasticity.

Attendees will gain a deep understanding of how vLLM implements next-generation parallelism strategies and what it takes to run cluster-scale MoE and non-MoE LLMs efficiently in real production environments.

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
How Red Hat Scales Large-Scale Serving with vLLM | Ray Summit 2025Ray in Production: Engineering Lessons from Spotifys Hendrix Framework | Spotify | Ray Summit 2026Accelerating Large Language Models: vLLM on TPUs | Google | Ray Summit 2026KubeSecRay: Fortifying Multi-Tenant Ray Clusters on Kubernetes | Ray Summit 2024Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025Webinar: Scaling LLM Fine-Tuning with FSDP, DeepSpeed, and RayFrom Prototype to Production: Securely Accelerating Physical AI with Vision-Language-Action ModelsClosing the Train-Inference Divide for Asynchronous RL at Scale | Lila Sciences | Ray Summit 2026SkyRL tx: A unified training and inference engine | Ray Summit 2025Scaling Production LLM Inference Using EKS Auto Mode & Ray Serve | Ray Summit 2025From Self-Managed Ray to Anyscale Services: Migrating ML Serving | Coinbase | Ray Summit 2026
Anyscale |

How Red Hat Scales Large-Scale Serving with vLLM | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER