How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025 @anyscale
How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025  @anyscale
Uploaded December 2025 | Updated November 2026, 1 hour ago
At Ray Summit 2025, Wenyue Liu and Akshit Trehan from Coinbase share how the Coinbase Machine Learning Platform (MLP) team built trusted, production-grade LLM services using Ray, vLLM, and LiteLLM—supporting one of the world’s most security-sensitive environments and reinforcing Coinbase’s mission to remain the most trusted crypto exchange.

They begin by outlining the unique challenges of building LLM infrastructure inside a financial institution, where trust, security, and reliability are non-negotiable. To meet these requirements, Coinbase engineered an LLM serving stack that seamlessly integrates:

Ray for distributed orchestration and scaling

vLLM for high-throughput, low-latency inference

LiteLLM for routing, abstraction, and multi-provider reliability

The speakers then take a deep dive into the technical architecture behind Coinbase’s internal LLM services, including:

User authentication and authorization patterns tailored for secure LLM access

Service-to-service (s2s) trust models that allow safe and auditable communication between internal systems

LiteLLM distribution strategies to balance throughput, reliability, and fallback behavior

How vLLM and Ray work together to power scalable, production-grade LLM serving APIs

Systems built to support high-volume internal LLM traffic, ensuring consistent performance under load

The session walks through the full end-to-end story of how Coinbase uses Ray and vLLM to deliver trustworthy, secure, and efficient LLM services—meeting the strict reliability requirements of a top global crypto exchange.

Liked this video? Check out other Ray Summit breakout session recordings youtube.com/playlist?list=PLzTswPQNepXllnU0C36WtkC0dqkAoDulh

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025MLOps with Ray on Anyscale | Ray Summit 2025Distributed Embeddings at Scale: Processing 10M+ Rows/ Day with Ray, GPUs & Qdrant | Ray Summit 2025Scaling Physical AI & Robotics Systems with Ray | Anyscale | Ray Summit 2026Serving vLLM on Intel GPUs, CPUs, and Gaudi | Intel | Ray Summit 2026Elastic Expert Parallelism for vLLM | Ray Summit 2025Post-training Nemotron with Ray | Bryan Catanzaro (Nvidia) | Ray Summit 2026Apple’s Approach to Scalable Machine Learning Infrastructure on Ray | Ray Summit 2025Serving Fast and Smart: Getting the Most Out of Ray Serve LLM on GKE | Google | Ray Summit 2026Anyscale on Azure: Build, Train, and Serve AI in Your Own Tenant | Microsoft | Ray Summit 2026Improved Scheduling Flexibility with Label Selectors in Ray | Ray Summit 2025Meet verl: An RL Framework for LLM Reasoning & Tool Use | Ray Summit 2025
Anyscale |

How Coinbase Uses Ray, vLLM & LiteLLM to Power Secure LLM Services | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER