Uploaded September 2026 | Updated November 2026, 3 hours ago
"Github repository: github.com/anyscale/ray-summit-training-2026/tree/main/ray_summit_distributed_inference_2026
Design scalable distributed inference systems for production AI applications, from classic ML models to large language models.
This hands-on training session covers building Ray Serve applications with deployments, composition, and load-aware request routing, hardening for production with autoscaling, fault tolerance, and zero-downtime upgrades, deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang, and optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching.
You'll leave knowing how to scale frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts models.
Liked this video? Check out other Ray Summit training session recordings: youtube.com/playlist?list=PLUpYVYRTFCQM
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com/"
"Github repository: github.com/anyscale/ray-summit-training-2026/tree/main/ray_summit_distributed_inference_2026
Design scalable distributed inference systems for production AI applications, from classic ML models to large language models.
This hands-on training session covers building Ray Serve applications with deployments, composition, and load-aware request routing, hardening for production with autoscaling, fault tolerance, and zero-downtime upgrades, deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang, and optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching.
You'll leave knowing how to scale frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts models.
Liked this video? Check out other Ray Summit training session recordings: youtube.com/playlist?list=PLUpYVYRTFCQM
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com/"










