Ray Serve: Advancing scalability and flexibility | Ray Summit 2025 @anyscale
Ray Serve: Advancing scalability and flexibility | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 16 minutes ago
At Ray Summit 2025, Abrar Sheikh and Alexander Yang from Anyscale share the major advancements in Ray Serve, now one of the most widely used libraries for powering modern AI applications across industries.

They begin by highlighting why Ray Serve stands apart from traditional online inference frameworks: it is natively built for multi-model serving, supports any hardware and accelerator, and integrates cleanly with any inference engine—from vLLM and TensorRT-LLM to custom model runtimes.

The session then dives into the most significant improvements Ray Serve has delivered over the past year, including:

Greater flexibility for complex inference patterns
Expanded APIs and routing capabilities make it easier to serve multi-stage pipelines, ensemble models, agentic systems, and inference graphs.

Higher performance at scale
Under-the-hood optimizations, improved scheduling, and faster data movement enable Ray Serve to handle massive request volumes with lower latency and higher throughput.

Multi-cloud inference support
New features make it easier to deploy Ray Serve clusters across multiple cloud providers—supporting hybrid inference, failover strategies, and portable deployment architectures.

Abrar and Alexander demonstrate how Ray Serve continues to evolve to meet the needs of cutting-edge AI systems—from LLMs to multimodal workloads—and offer a look at what's next for the framework.

Attendees will learn how to take advantage of Ray Serve’s newest capabilities to build flexible, performant, and cloud-agnostic inference platforms at scale.

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Ray Serve: Advancing scalability and flexibility | Ray Summit 2025Trillion-Token Pretraining | Adyen | Ray Summit 2026Running Ray in Production: Google’s Guide to Operators & Observability | Ray Summit 2025From Agents to Action: Scaling Agentic AI with Ray on Intel Xeon | Intel | Ray Summit 2026Building an AI Science Factory | Andrew Beam (Lila Sciences) | Ray Summit 2026GLM-5 RL Training with vLLM | Prime Intellect | Ray Summit 2026Accelerating vLLM with LMCache | Ray Summit 2025Ray History Server: Observability for Ephemeral K8s Clusters | KubeRay | Ray Summit 2026SQL or Python? Why It’s the Wrong Question in the AI Era | Ray Summit 2025Brendan Burns: Lessons from Building Kubernetes and the Future of AI InfrastructureScaling AI the Snowflake Way: ML Workloads on Ray | Ray Summit 2025Scaling Ray on Kubernetes: Pragmatic Strategies for Every Team | Ray Summit 2025
Anyscale |

Ray Serve: Advancing scalability and flexibility | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER