How AWS Scales Reinforcement Learning Across Thousands of GPUs | Ray Summit 2025 @anyscale
How AWS Scales Reinforcement Learning Across Thousands of GPUs | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 19 hours ago
At Ray Summit 2025, Kunal Jha and Anoop Saha from AWS share how to build a fault-tolerant, scalable RL factory for large-scale LLM alignment by combining the elasticity of Ray with the resiliency of Amazon SageMaker HyperPod.

They begin by outlining the new reality of reinforcement learning: as RL expands beyond gaming and robotics into LLM alignment, agentic workloads, and real-world control with trillion-parameter base models, success depends not just on raw speed but on robustness, scalability, and elasticity. Many RL pipelines fail at cluster scale—not because GPUs are insufficient, but because GPU failures, preemptions, and tail latencies degrade goodput, the true measure of useful learning per GPU-hour.

Ray’s unified programming model enables RL pipelines to scale seamlessly from a single machine to massive distributed clusters, and is already powering popular frameworks like Verl, which rely on Ray to spin up workers, orchestrate rollouts, and move data efficiently across nodes.

The speakers then introduce Amazon SageMaker HyperPod, a persistent, highly resilient GPU cluster designed for distributed AI at extreme scale. By pairing Ray’s efficiency with HyperPod’s robustness, teams can build scalable post-training pipelines capable of running reliably across hundreds or thousands of GPUs.

Kunal and Anoop dive into the architectural details of:

Running Ray Jobs on HyperPod at massive scale

Leveraging vLLM for high-throughput inference workers

Coordinating PPO, GRPO, and DAPO pipelines using Verl + Ray

Recovering gracefully from GPU failures and preemptions to maximize goodput

Designing an RL factory architecture suitable for trillion-parameter model alignment

They also share reference implementations for post-training large open-weight models, along with practical strategies for optimizing performance, cost, and reliability.

Attendees will learn how Ray and Amazon SageMaker HyperPod together enable a new class of fault-tolerant, massively scalable RL systems for next-generation LLM alignment.

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
How AWS Scales Reinforcement Learning Across Thousands of GPUs | Ray Summit 2025Distributed Model Training with Ray at Capital One | Ray Summit 2025State of vLLM 2026 | Inferact | Ray Summit 2026From Benchmark #1 to Production | Nebius | Ray Summit 2026Foundation Model Distributed Training with Ray | Anyscale | Ray Summit 2026High-Throughput Inference for Synthetic Data & Evals at Sutro | Ray Summit 2025How a Transformers Model Loads in vLLM | Hugging Face | Ray Summit 2026NVIDIA’s Framework for Scalable Data Curation | Ray Summit 2025Koba: Disaggregated Data Loading for Multimodal Training at Luma AI | Luma AI | Ray Summit 2026A Compiler for Christmas | Brendan Burns #raysummitTowards Clinically Grounded LLMs | SonderMind | Ray Summit 2026Scaling LLM Post-Training at Character.AI | Ray Summit 2025
Anyscale |

How AWS Scales Reinforcement Learning Across Thousands of GPUs | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER