Uploaded September 2026 | Updated November 2026, 4 days ago
As Ray Serve deployments grow to thousands of autoscaling replicas, the Serve controller, the brain coordinating every replica and routing decision, becomes the bottleneck: control-loop latency climbs and autoscaling decisions act on stale, seconds-old metrics.
At Ray Summit 2026, Abrar Sheikh, Software Engineer at Anyscale, walks through a systematic scalability experiment that pushed a single controller to 2,048 replicas and pinpointed where the time actually goes, plus the two optimizations that moved the needle: a Pydantic v2 migration of the serialization layer and a Cython-based metrics aggregation path, together delivering a 7x faster control loop, 3.6x higher controller throughput, and 15x fresher replica metrics at peak scale.
You'll leave knowing where controller time goes and how to scale past it.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
As Ray Serve deployments grow to thousands of autoscaling replicas, the Serve controller, the brain coordinating every replica and routing decision, becomes the bottleneck: control-loop latency climbs and autoscaling decisions act on stale, seconds-old metrics.
At Ray Summit 2026, Abrar Sheikh, Software Engineer at Anyscale, walks through a systematic scalability experiment that pushed a single controller to 2,048 replicas and pinpointed where the time actually goes, plus the two optimizations that moved the needle: a Pydantic v2 migration of the serialization layer and a Cython-based metrics aggregation path, together delivering a 7x faster control loop, 3.6x higher controller throughput, and 15x fresher replica metrics at peak scale.
You'll leave knowing where controller time goes and how to scale past it.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










