Uploaded November 2025 | Updated November 2026, 12 hours ago
At Ray Summit 2025, Nisha Mariam Johnson and Ryan O'Leary from Google share how to build an efficient, cost-effective, and accelerator-aware ML platform using Ray on Kubernetes—designed to tackle challenges in scheduling, cloud costs, and emerging hardware.
They begin by outlining how teams can establish a smarter foundation for resource management. This includes mastering GKE Custom Compute Classes and label-based scheduling to seamlessly shift workloads across spot instances, diverse GPU and TPU types, and reserved capacity—dramatically reducing cloud costs while maintaining strong performance. They also cover how GKE image streaming speeds up image and model loading to reduce startup latency.
Next, Nisha and Ryan explain how fair-share scheduling is achieved with Kueue, Kubernetes’ native job-queueing system. Integrated directly with Ray, Kueue ensures teams receive fair access to compute resources for both training and serving workloads, preventing starvation and improving reliability across shared clusters.
With this foundation in place, the speakers dive into how these capabilities support high-value ML workloads:
Large-scale training: Learn how to train on TPUs using Ray Train with familiar frameworks like JAX and PyTorch, and how integrated TPU metrics in the Ray Dashboard make performance debugging dramatically simpler.
High-throughput serving and batch inference: See how workloads can be dynamically shifted between GPUs and TPUs to optimize for both cost and latency. They also preview upcoming Ray features that further enhance large-scale training, high-throughput inference, and complex RL and agentic workloads—all running on a secure, decoupled GKE architecture.
Attendees will leave with a clear blueprint for the recommended architecture for Ray on Kubernetes, unlocking new levels of performance, cost-efficiency, and operational simplicity for ML workloads of any scale.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
At Ray Summit 2025, Nisha Mariam Johnson and Ryan O'Leary from Google share how to build an efficient, cost-effective, and accelerator-aware ML platform using Ray on Kubernetes—designed to tackle challenges in scheduling, cloud costs, and emerging hardware.
They begin by outlining how teams can establish a smarter foundation for resource management. This includes mastering GKE Custom Compute Classes and label-based scheduling to seamlessly shift workloads across spot instances, diverse GPU and TPU types, and reserved capacity—dramatically reducing cloud costs while maintaining strong performance. They also cover how GKE image streaming speeds up image and model loading to reduce startup latency.
Next, Nisha and Ryan explain how fair-share scheduling is achieved with Kueue, Kubernetes’ native job-queueing system. Integrated directly with Ray, Kueue ensures teams receive fair access to compute resources for both training and serving workloads, preventing starvation and improving reliability across shared clusters.
With this foundation in place, the speakers dive into how these capabilities support high-value ML workloads:
Large-scale training: Learn how to train on TPUs using Ray Train with familiar frameworks like JAX and PyTorch, and how integrated TPU metrics in the Ray Dashboard make performance debugging dramatically simpler.
High-throughput serving and batch inference: See how workloads can be dynamically shifted between GPUs and TPUs to optimize for both cost and latency. They also preview upcoming Ray features that further enhance large-scale training, high-throughput inference, and complex RL and agentic workloads—all running on a secure, decoupled GKE architecture.
Attendees will leave with a clear blueprint for the recommended architecture for Ray on Kubernetes, unlocking new levels of performance, cost-efficiency, and operational simplicity for ML workloads of any scale.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










