Serving LLMs for Agents at Scale with Ray + vLLM | JPMorgan Chase | Ray Summit 2026 @anyscale
Serving LLMs for Agents at Scale with Ray + vLLM | JPMorgan Chase | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 22 hours ago
Serving LLMs for agentic workloads at Chase scale means balancing latency, cost, and resilience. The team cut latency from 2,400ms to 275ms, roughly 8.7x faster, with self-hosted LLMs running about 10x faster and 12x cheaper than third-party providers.

At Ray Summit 2026, Ankur Mohan and Dhayaneshwar Balusamy from JPMorgan Chase show how they built the production platform powering the Chase Digital Assistant across multiple domain agents on Ray and vLLM: a multi-agent architecture with request routing that directs each request to the appropriate model, custom inference optimizations, detailed profiling, and hardware experiments across H100 and B200 GPUs.

You'll leave with a blueprint for self-hosting LLMs for agentic workloads at enterprise scale.

Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Serving LLMs for Agents at Scale with Ray + vLLM | JPMorgan Chase | Ray Summit 2026Lessons from Unifying Simulation and RL Training | Torc Robotics | Ray Summit 2026Ray Train: Distributed Solutions for Removing Training Bottlenecks | Ray Summit 2025Building a Multimodal Video Processing Pipeline with RayHow Open-Source vLLM Topped the Artificial Analysis Leaderboard | DigitalOcean | Ray Summit 2026How DigitalOcean Builds Next-Gen Inference with Ray, vLLM & More | Ray Summit 2025Leveraging Agent Skills for Ray Workloads | Anyscale | Ray Summit 2026Serving Frontier MoE Models at the Lowest Token Cost | NVIDIA | Ray Summit 2026NVIDIA NeMo Curator: Scaling Multi-Modal Data Curation Workflows | Ray Summit 2025How Daft Boosts Batch Inference Throughput with Dynamic Partitioning | Ray Summit 2025Wisedocs’ Journey: Rebuilding & Accelerating ML with KubeRay | Ray Summit 2025AMD and vLLM: Whats New | AMD | Ray Summit 2026
Anyscale |

Serving LLMs for Agents at Scale with Ray + vLLM | JPMorgan Chase | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER