Uploaded November 2025 | Updated November 2026, 1 day ago
At Ray Summit 2025, Deepak Chandramouli, Rehan Durrani, and Ankur Goenka from Apple share how they built an internal, enterprise-grade serverless model-hosting platform powered by Ray Serve and vLLM, enabling fast, scalable LLM inference across teams.
They begin by outlining the design of their self-service platform, inspired by best-in-class industry solutions but tailored for Apple’s internal developer ecosystem. The platform allows teams to deploy and manage models seamlessly, while abstracting away the underlying operational complexity.
The speakers then dive into the critical capabilities layered on top of Ray Serve and vLLM, including:
Robust multi-tenancy to isolate workloads and enforce organizational boundaries
Dynamic autoscaling to handle unpredictable real-world traffic patterns
Token-level budgeting and metering to enforce usage constraints and cost transparency
Deep request-level observability for debugging, monitoring, and performance tuning
Fine-grained resource controls to optimize cluster utilization
They also discuss the architectural challenges the team faced—and the solutions they developed—while making LLM inference reliable, efficient, and secure for enterprise environments.
Attendees will learn how Ray Serve and vLLM can be combined to build a production-grade model serving platform, whether for internal developers or external customers, and walk away with practical patterns for operating LLM inference at scale.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
At Ray Summit 2025, Deepak Chandramouli, Rehan Durrani, and Ankur Goenka from Apple share how they built an internal, enterprise-grade serverless model-hosting platform powered by Ray Serve and vLLM, enabling fast, scalable LLM inference across teams.
They begin by outlining the design of their self-service platform, inspired by best-in-class industry solutions but tailored for Apple’s internal developer ecosystem. The platform allows teams to deploy and manage models seamlessly, while abstracting away the underlying operational complexity.
The speakers then dive into the critical capabilities layered on top of Ray Serve and vLLM, including:
Robust multi-tenancy to isolate workloads and enforce organizational boundaries
Dynamic autoscaling to handle unpredictable real-world traffic patterns
Token-level budgeting and metering to enforce usage constraints and cost transparency
Deep request-level observability for debugging, monitoring, and performance tuning
Fine-grained resource controls to optimize cluster utilization
They also discuss the architectural challenges the team faced—and the solutions they developed—while making LLM inference reliable, efficient, and secure for enterprise environments.
Attendees will learn how Ray Serve and vLLM can be combined to build a production-grade model serving platform, whether for internal developers or external customers, and walk away with practical patterns for operating LLM inference at scale.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










