Embedded LLM’s Guide to vLLM Architecture & High-Performance Serving | Ray Summit 2025 @anyscale
Embedded LLM’s Guide to vLLM Architecture & High-Performance Serving | Ray Summit 2025  @anyscale
Uploaded November 2025 | Updated November 2026, 18 hours ago
At Ray Summit 2025, Tun Jian Tan from Embedded LLM shares an inside look at what gives vLLM its industry-leading speed, flexibility, and extensibility—offering a deep architectural walkthrough designed for everyone from day-to-day users to future kernel-level contributors.

The session uses a real-world case study—integrating AITER (AI Tensor Engine for ROCm) kernels and Multi-head Latent Attention (MLA) to accelerate DeepSeek-R1 inference on AMD GPUs—to illuminate how vLLM’s internals work and how new hardware and models plug into the system.

The talk is structured to deliver value across experience levels:

For vLLM Users & MLOps Engineers:
Gain a strong mental model of vLLM’s core architecture. Tun Jian explores internal components such as CustomOps, attention mechanisms, and the integration points for high-performance kernels—showing how vLLM achieves high throughput and rapidly incorporates new model architectures and hardware backends.

For Kernel Developers & Contributors:
Learn exactly how to contribute effectively to vLLM. The session maps out where new logic should live, how the Python API bridges to custom kernels, and common pitfalls—such as custom ops not being invoked without proper registration. Debugging workflows (e.g., tracing CPU overhead) and benchmarking practices are covered to help contributors turn kernel prototypes into production-ready pull requests.

Whether you're optimizing production deployments or preparing to write the next major vLLM feature, this session will show you what makes vLLM tick—and how you can tap into its architecture to push LLM inference even further.

Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
Embedded LLM’s Guide to vLLM Architecture & High-Performance Serving | Ray Summit 2025Ray Agent Engine: Deploying AI Agents with Ray Serve | Ray Summit 2025Planetary-scale multimodal image processing | Xoople + AnyscalePlatform Orchestration: Anyscale + Ray on Kubernetes | Anyscale | Ray Summit 2026How DataRobot Parallelizes Agentic Pipeline Searches with Ray + syftr | Ray Summit 2025Diffusion Large Language Models | Inception Labs | Ray Summit 2026Efficient, High-Performance AI Inferencing with Intel Xeon 6 | Ray Summit 2025Cascade: Write-Once Run-Anywhere Inference on Ray | Samsara | Ray Summit 2026Maximizing GB200 and GB300 Performance with Domain-Aware Scheduling | NVIDIA | Ray Summit 2026Ray Data on GPU - Breaking Preprocessing Bottleneck for Model Training | Stripe | Ray Summit 2026Miles: Enterprise-Facing Agentic RL Framework | RadixArk + Anyscale | Ray Summit 2026Beyond Text-to-Video: Omni-Context Generation in a 33B Model | MiniMax | Ray Summit 2026
Anyscale |

Embedded LLM’s Guide to vLLM Architecture & High-Performance Serving | Ray Summit 2025

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER