Uploaded September 2026 | Updated November 2026, 22 hours ago
Serving LLMs for agentic workloads at Chase scale means balancing latency, cost, and resilience. The team cut latency from 2,400ms to 275ms, roughly 8.7x faster, with self-hosted LLMs running about 10x faster and 12x cheaper than third-party providers.
At Ray Summit 2026, Ankur Mohan and Dhayaneshwar Balusamy from JPMorgan Chase show how they built the production platform powering the Chase Digital Assistant across multiple domain agents on Ray and vLLM: a multi-agent architecture with request routing that directs each request to the appropriate model, custom inference optimizations, detailed profiling, and hardware experiments across H100 and B200 GPUs.
You'll leave with a blueprint for self-hosting LLMs for agentic workloads at enterprise scale.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Serving LLMs for agentic workloads at Chase scale means balancing latency, cost, and resilience. The team cut latency from 2,400ms to 275ms, roughly 8.7x faster, with self-hosted LLMs running about 10x faster and 12x cheaper than third-party providers.
At Ray Summit 2026, Ankur Mohan and Dhayaneshwar Balusamy from JPMorgan Chase show how they built the production platform powering the Chase Digital Assistant across multiple domain agents on Ray and vLLM: a multi-agent architecture with request routing that directs each request to the appropriate model, custom inference optimizations, detailed profiling, and hardware experiments across H100 and B200 GPUs.
You'll leave with a blueprint for self-hosting LLMs for agentic workloads at enterprise scale.
Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo
Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale
đź”— Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










