Production-Ready Distributed Inference with Ray Serve | Anyscale | Ray Summit 2026 @anyscale
Production-Ready Distributed Inference with Ray Serve | Anyscale | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 3 hours ago
"Github repository: github.com/anyscale/ray-summit-training-2026/tree/main/ray_summit_distributed_inference_2026

Design scalable distributed inference systems for production AI applications, from classic ML models to large language models.

This hands-on training session covers building Ray Serve applications with deployments, composition, and load-aware request routing, hardening for production with autoscaling, fault tolerance, and zero-downtime upgrades, deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang, and optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching.

You'll leave knowing how to scale frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts models.

Liked this video? Check out other Ray Summit training session recordings: youtube.com/playlist?list=PLUpYVYRTFCQM

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com/"
Production-Ready Distributed Inference with Ray Serve | Anyscale | Ray Summit 2026Petabyte-Scale Video Curation on CoreWeave | CoreWeave | Ray Summit 2026Ray Direct Transport: RDMA Support in Ray Core | Ray Summit 2025Maximizing Compute Efficiency on Anyscale | Ray Summit 2025The PARK Stack: The LAMP Stack of the AI Era | Ben LoricaProduction-ready Distributed Inference with Ray ServeAgentic Workload Inference at Scale: ByteDance’s AIBrix & DeerFlow | Ray Summit 2025Real-Time Search & Recommendation Systems for AI Commerce | Anyscale | Ray Summit 2026Exabyte-scale Streaming Iceberg IO with Ray, Flink, and DeltaCAT | Ray Summit 2025End-to-End GenAI Orchestration with KubeRay | Ray Summit 2025Fixing S3 Bottlenecks: Scalable I/O for Ray with Alluxio | Ray Summit 2025Scaling Post-Training Workflows with Ray Data, Ray Data LLM, and vLLM | Ray Summit 2025
Anyscale |

Production-Ready Distributed Inference with Ray Serve | Anyscale | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER