Scaling Ray on Kubernetes: Pragmatic Strategies for Every Team | Ray Summit 2025 @anyscale
Scaling Ray on Kubernetes: Pragmatic Strategies for Every Team | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 11 hours ago
At Ray Summit 2025, Spencer Peterson and Raja Jadeja from Google share the definitive playbook for using KubeRay to build a stable, efficient, and self-service AI platform on Kubernetes—eliminating the operational chaos caused by ad-hoc ML deployments, YAML overload, and runaway cluster resource usage.

They begin by calling out the common challenges faced by ML infrastructure teams:

Unschedulable pods when a single rogue job consumes all available resources

Idle GPUs inflating cloud bills

Developers trapped in complex Kubernetes configs just to scale a Python script

Manual deployments that are brittle, slow, and impossible to manage at scale

KubeRay solves these issues by transforming Kubernetes into an application-aware backend for distributed AI. With the RayCluster custom resource, Kubernetes can natively manage Ray applications end-to-end—from creation and scheduling to autoscaling, fault tolerance, and cleanup.

Spencer and Raja walk through the operational strategies needed to run production Ray clusters effectively:

Tame the noisy neighbor problem: Implement true multi-tenancy using namespace isolation and cgroups to ensure resource fairness.

Master workload placement: Use Ray’s placement features and Kubernetes labels to automatically route jobs to the optimal nodes—maximizing efficiency without requiring deep Kubernetes expertise.

Cut cloud costs with smart scaling: Configure IPPR with the Ray v2 autoscaler for faster pod scheduling and on-demand scaling, eliminating costly over-provisioning.

Run a unified multi-cluster platform: Learn best practices for managing multiple specialized Ray clusters under one Kubernetes control plane.

Attendees will leave with a clear, actionable blueprint for turning Kubernetes into a robust, cost-efficient, developer-friendly platform for distributed AI—powered by KubeRay.

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Scaling Ray on Kubernetes: Pragmatic Strategies for Every Team | Ray Summit 2025The Rise of vLLM: Building an Open Source LLM Inference EngineHow Red Hat Scales Large-Scale Serving with vLLM | Ray Summit 2025Ray in Production: Engineering Lessons from Spotifys Hendrix Framework | Spotify | Ray Summit 2026Accelerating Large Language Models: vLLM on TPUs | Google | Ray Summit 2026KubeSecRay: Fortifying Multi-Tenant Ray Clusters on Kubernetes | Ray Summit 2024Scaling the Ray Serve Controller to Thousands of Replicas | Anyscale | Ray Summit 2026Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025Webinar: Scaling LLM Fine-Tuning with FSDP, DeepSpeed, and RayFrom Prototype to Production: Securely Accelerating Physical AI with Vision-Language-Action ModelsClosing the Train-Inference Divide for Asynchronous RL at Scale | Lila Sciences | Ray Summit 2026SkyRL tx: A unified training and inference engine | Ray Summit 2025
Anyscale |

Scaling Ray on Kubernetes: Pragmatic Strategies for Every Team | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER