Scaling LLMs at Apple: Ray Serve + vLLM Deep Dive | Ray Summit 2025 @anyscale
Scaling LLMs at Apple: Ray Serve + vLLM Deep Dive | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 1 day ago
At Ray Summit 2025, Deepak Chandramouli, Rehan Durrani, and Ankur Goenka from Apple share how they built an internal, enterprise-grade serverless model-hosting platform powered by Ray Serve and vLLM, enabling fast, scalable LLM inference across teams.

They begin by outlining the design of their self-service platform, inspired by best-in-class industry solutions but tailored for Apple’s internal developer ecosystem. The platform allows teams to deploy and manage models seamlessly, while abstracting away the underlying operational complexity.

The speakers then dive into the critical capabilities layered on top of Ray Serve and vLLM, including:

Robust multi-tenancy to isolate workloads and enforce organizational boundaries
Dynamic autoscaling to handle unpredictable real-world traffic patterns
Token-level budgeting and metering to enforce usage constraints and cost transparency
Deep request-level observability for debugging, monitoring, and performance tuning
Fine-grained resource controls to optimize cluster utilization

They also discuss the architectural challenges the team faced—and the solutions they developed—while making LLM inference reliable, efficient, and secure for enterprise environments.

Attendees will learn how Ray Serve and vLLM can be combined to build a production-grade model serving platform, whether for internal developers or external customers, and walk away with practical patterns for operating LLM inference at scale.

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Scaling LLMs at Apple: Ray Serve + vLLM Deep Dive | Ray Summit 2025Scaling Robot Policy Evaluations to Thousands of Parallel SimulationsWhat Makes RL the Hardest Workload in AI | Robert Nishihara (Anyscale) | Ray Summit 2026Ray Data: Whats New - GPU Shuffle, Unity Catalog, DataSourceV2 | Anyscale | Ray Summit 2026Inside Adobe Firefly: JIT-Embedding with Ray Serve for Faster GenAI Training | Ray Summit 2025Running Embedding Processing for AI Search at Scale | Notion + AnyscaleServing vLLM on Agentic Production Workloads | Inferact | Ray Summit 2026Cost-Effective RL Post-Training with Ray and AWS Trainium | AWS + Anyscale | Ray Summit 2026Ray Serve: Advancing scalability and flexibility | Ray Summit 2025Trillion-Token Pretraining | Adyen | Ray Summit 2026Running Ray in Production: Google’s Guide to Operators & Observability | Ray Summit 2025From Agents to Action: Scaling Agentic AI with Ray on Intel Xeon | Intel | Ray Summit 2026
Anyscale |

Scaling LLMs at Apple: Ray Serve + vLLM Deep Dive | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER