Custom Pipeline Parallelism for Frontier-Scale Model Serving | Apple | Ray Summit 2026 @anyscale
Custom Pipeline Parallelism for Frontier-Scale Model Serving | Apple | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 3 days ago
Serving frontier-scale mixture-of-experts models at low latency requires cross-node pipeline parallelism, but vLLM's compiled-DAG approach deadlocks beyond two stages, and the safe workaround leaves half the GPUs idle.

At Ray Summit 2026, Abin Shahab and Michael Lee from Apple show how they turned Ray's single-threaded FIFO actor mailboxes into the pipeline scheduler itself, bypassing compiled DAGs, then cut two hidden CPU bottlenecks to reach up to 1.5x throughput.

You'll leave able to diagnose these failures and rebuild fast pipeline parallelism on Ray and vLLM.

Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Custom Pipeline Parallelism for Frontier-Scale Model Serving | Apple | Ray Summit 2026Scaling LinkedIns Online Training Solution with Ray | Ray Summit 2025AI workloads introduce a system design shift ...Inside Lilas AI Research Platform | Lila Sciences | Ray Summit 2026LLM Post-Training and High-Performance Serving | Anyscale | Ray Summit 2026Training Frontier Knowledge Work Agents: A 397B Open Recipe with SkyRL | Mercor | Ray Summit 2026Building Self-Serve, Multimodal Medical Image Pipelines with Ray | Heartflow | Ray Summit 2026Agents and Robots: The Human Effort Putting AI Into Production | Encord | Ray Summit 2026How Exa Indexes the Entire Web: Ray Data and Lance at Billion-Document Scale | Exa | Ray Summit 2026Unlocking Peak Workload Performance & Efficiency with Ray on Kubernetes | Ray Summit 2025Boosting vLLM Inference on Huawei NPU with Ray Compiled Graphs — Huawei | Ray Summit 2025Simulating the Physical World at Scale: Humanoid Locomotion with Ray | Lambda | Ray Summit 2026
Anyscale |

Custom Pipeline Parallelism for Frontier-Scale Model Serving | Apple | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER