Uploaded November 2025 | Updated November 2026, 18 hours ago
Slides: drive.google.com/file/d/1jmA5vKn_mKl6qgFQdGBd0mnTNBGOLU9y/view?usp=sharing
At Ray Summit 2025, Suman Debnath and Linda Haviv from Anyscale share how to master distributed training strategies essential for efficiently scaling today’s deep learning models.
They begin by breaking down the core techniques—data parallelism, model parallelism, and pipeline parallelism—and explain when each approach is most effective as models and datasets grow. The session covers advanced methods such as sharded training and ZeRO, along with the tradeoffs that arise in real-world large-cluster environments.
Suman and Linda also address the toughest challenges in distributed training, including communication overhead, fault tolerance, reproducibility, and managing heterogeneous compute. They then demonstrate how PyTorch and Ray can be combined to implement these strategies with minimal code changes, making it easier to scale from prototype to production.
What you’ll learn:
How data, model, and pipeline parallelism work—and when to apply each
How to overcome scalability bottlenecks such as communication overhead and system failures
How to use Ray with PyTorch to launch, orchestrate, and monitor large-scale distributed training jobs
Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Slides: drive.google.com/file/d/1jmA5vKn_mKl6qgFQdGBd0mnTNBGOLU9y/view?usp=sharing
At Ray Summit 2025, Suman Debnath and Linda Haviv from Anyscale share how to master distributed training strategies essential for efficiently scaling today’s deep learning models.
They begin by breaking down the core techniques—data parallelism, model parallelism, and pipeline parallelism—and explain when each approach is most effective as models and datasets grow. The session covers advanced methods such as sharded training and ZeRO, along with the tradeoffs that arise in real-world large-cluster environments.
Suman and Linda also address the toughest challenges in distributed training, including communication overhead, fault tolerance, reproducibility, and managing heterogeneous compute. They then demonstrate how PyTorch and Ray can be combined to implement these strategies with minimal code changes, making it easier to scale from prototype to production.
What you’ll learn:
How data, model, and pipeline parallelism work—and when to apply each
How to overcome scalability bottlenecks such as communication overhead and system failures
How to use Ray with PyTorch to launch, orchestrate, and monitor large-scale distributed training jobs
Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










