How to Get Started with Distributed Training at Scale | Ray Summit 2025 @anyscale
How to Get Started with Distributed Training at Scale | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 18 hours ago
Slides: drive.google.com/file/d/1jmA5vKn_mKl6qgFQdGBd0mnTNBGOLU9y/view?usp=sharing

At Ray Summit 2025, Suman Debnath and Linda Haviv from Anyscale share how to master distributed training strategies essential for efficiently scaling today’s deep learning models.

They begin by breaking down the core techniques—data parallelism, model parallelism, and pipeline parallelism—and explain when each approach is most effective as models and datasets grow. The session covers advanced methods such as sharded training and ZeRO, along with the tradeoffs that arise in real-world large-cluster environments.

Suman and Linda also address the toughest challenges in distributed training, including communication overhead, fault tolerance, reproducibility, and managing heterogeneous compute. They then demonstrate how PyTorch and Ray can be combined to implement these strategies with minimal code changes, making it easier to scale from prototype to production.

What you’ll learn:

How data, model, and pipeline parallelism work—and when to apply each

How to overcome scalability bottlenecks such as communication overhead and system failures

How to use Ray with PyTorch to launch, orchestrate, and monitor large-scale distributed training jobs

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
How to Get Started with Distributed Training at Scale | Ray Summit 2025Benchmarking GPU Scheduling for Massive-Scale Ray Workloads at Minimal Cost - MSFT | Ray Summit 2025Embedded LLM’s Guide to vLLM Architecture & High-Performance Serving | Ray Summit 2025Ray Agent Engine: Deploying AI Agents with Ray Serve | Ray Summit 2025Planetary-scale multimodal image processing | Xoople + AnyscalePlatform Orchestration: Anyscale + Ray on Kubernetes | Anyscale | Ray Summit 2026How DataRobot Parallelizes Agentic Pipeline Searches with Ray + syftr | Ray Summit 2025Diffusion Large Language Models | Inception Labs | Ray Summit 2026Efficient, High-Performance AI Inferencing with Intel Xeon 6 | Ray Summit 2025Cascade: Write-Once Run-Anywhere Inference on Ray | Samsara | Ray Summit 2026Maximizing GB200 and GB300 Performance with Domain-Aware Scheduling | NVIDIA | Ray Summit 2026Ray Data on GPU - Breaking Preprocessing Bottleneck for Model Training | Stripe | Ray Summit 2026
Anyscale |

How to Get Started with Distributed Training at Scale | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER