Uploaded September 2026 | Updated November 2026, 3 days ago
Serving frontier-scale mixture-of-experts models at low latency requires cross-node pipeline parallelism, but vLLM's compiled-DAG approach deadlocks beyond two stages, and the safe workaround leaves half the GPUs idle.
At Ray Summit 2026, Abin Shahab and Michael Lee from Apple show how they turned Ray's single-threaded FIFO actor mailboxes into the pipeline scheduler itself, bypassing compiled DAGs, then cut two hidden CPU bottlenecks to reach up to 1.5x throughput.
You'll leave able to diagnose these failures and rebuild fast pipeline parallelism on Ray and vLLM.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Serving frontier-scale mixture-of-experts models at low latency requires cross-node pipeline parallelism, but vLLM's compiled-DAG approach deadlocks beyond two stages, and the safe workaround leaves half the GPUs idle.
At Ray Summit 2026, Abin Shahab and Michael Lee from Apple show how they turned Ray's single-threaded FIFO actor mailboxes into the pipeline scheduler itself, bypassing compiled DAGs, then cut two hidden CPU bottlenecks to reach up to 1.5x throughput.
You'll leave able to diagnose these failures and rebuild fast pipeline parallelism on Ray and vLLM.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










