Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026 @anyscale
Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 4 days ago
As Ray Serve deployments grow to thousands of autoscaling replicas, the Serve controller, the brain coordinating every replica and routing decision, becomes the bottleneck: control-loop latency climbs and autoscaling decisions act on stale, seconds-old metrics.

At Ray Summit 2026, Abrar Sheikh, Software Engineer at Anyscale, walks through a systematic scalability experiment that pushed a single controller to 2,048 replicas and pinpointed where the time actually goes, plus the two optimizations that moved the needle: a Pydantic v2 migration of the serialization layer and a Cython-based metrics aggregation path, together delivering a 7x faster control loop, 3.6x higher controller throughput, and 15x fresher replica metrics at peak scale.

You'll leave knowing where controller time goes and how to scale past it.

Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025Webinar: Scaling LLM Fine-Tuning with FSDP, DeepSpeed, and RayFrom Prototype to Production: Securely Accelerating Physical AI with Vision-Language-Action ModelsClosing the Train-Inference Divide for Asynchronous RL at Scale | Lila Sciences | Ray Summit 2026SkyRL tx: A unified training and inference engine | Ray Summit 2025Scaling Production LLM Inference Using EKS Auto Mode & Ray Serve | Ray Summit 2025From Self-Managed Ray to Anyscale Services: Migrating ML Serving | Coinbase | Ray Summit 2026Where’s My Training Job? Ray & GPU Observability at Datadog | Datadog | Ray Summit 2026Serving a Frontier Open Model with Ray & vLLM | Nscale | Ray Summit 2026Multi-Region, Environment Isolated Ray Environments in the Notion Way | Notion | Ray Summit 2026Pinterest’s Approach to Real-Time ML Experimentation Using Ray | Ray Summit 2025
Anyscale |

Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER