Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025 @anyscale
Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025  @anyscale
Uploaded December 2025 | Updated November 2026, 2 days ago
At Ray Summit 2025, Savita Manghnani from Datadog shares how Datadog keeps large-scale Ray training workloads fast, reliable, and observable—turning opaque, multi-GPU distributed jobs into systems engineers can monitor, debug, and optimize with confidence.

She begins by outlining the messy realities of training LLMs and AI models at scale: thousands of tasks running across heterogeneous GPU clusters, jobs stalling without clear causes, GPUs going idle unexpectedly, and slowdowns that are difficult to attribute. While Ray provides the distributed execution foundation, effective observability is what makes large-scale workloads predictable and operable.

Savita then dives into the real-world challenges Datadog encounters when running Ray internally to power LLM training, along with the observability techniques the team developed to address them. She covers:

Common failure modes in large Ray deployments—task backpressure, resource fragmentation, object store contention, spillover, slow nodes, and scheduler bottlenecks

Which signals matter most, including the specific metrics, traces, and logs that actually help diagnose bottlenecks and failures

How correlating these signals transforms Ray training from a “black box” into a transparent, debuggable system

Techniques Datadog uses to identify GPU underutilization, track straggler tasks, analyze scheduling delays, and detect systemic issues before they escalate

Finally, Savita shows how these observability practices make distributed AI workloads more reliable, explainable, and performant, whether you're training your first multi-node model or operating production-scale LLM jobs.

Attendees will walk away with actionable strategies for building observability into their Ray clusters—and for turning large distributed training jobs into systems they can truly understand and improve.

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025Webinar: Scaling LLM Fine-Tuning with FSDP, DeepSpeed, and RayFrom Prototype to Production: Securely Accelerating Physical AI with Vision-Language-Action ModelsClosing the Train-Inference Divide for Asynchronous RL at Scale | Lila Sciences | Ray Summit 2026SkyRL tx: A unified training and inference engine | Ray Summit 2025Scaling Production LLM Inference Using EKS Auto Mode & Ray Serve | Ray Summit 2025From Self-Managed Ray to Anyscale Services: Migrating ML Serving | Coinbase | Ray Summit 2026Where’s My Training Job? Ray & GPU Observability at Datadog | Datadog | Ray Summit 2026Serving a Frontier Open Model with Ray & vLLM | Nscale | Ray Summit 2026Multi-Region, Environment Isolated Ray Environments in the Notion Way | Notion | Ray Summit 2026Pinterest’s Approach to Real-Time ML Experimentation Using Ray | Ray Summit 2025From OOMs to 2x Throughput: Scaling Multimodal Data Curation on Ray | Zoox | Ray Summit 2026
Anyscale |

Taming Distributed AI Training with Ray + Datadog Observability | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER