Uploaded November 2025 | Updated November 2026, 18 hours ago
At Ray Summit 2025, Tun Jian Tan from Embedded LLM shares an inside look at what gives vLLM its industry-leading speed, flexibility, and extensibility—offering a deep architectural walkthrough designed for everyone from day-to-day users to future kernel-level contributors.
The session uses a real-world case study—integrating AITER (AI Tensor Engine for ROCm) kernels and Multi-head Latent Attention (MLA) to accelerate DeepSeek-R1 inference on AMD GPUs—to illuminate how vLLM’s internals work and how new hardware and models plug into the system.
The talk is structured to deliver value across experience levels:
For vLLM Users & MLOps Engineers:
Gain a strong mental model of vLLM’s core architecture. Tun Jian explores internal components such as CustomOps, attention mechanisms, and the integration points for high-performance kernels—showing how vLLM achieves high throughput and rapidly incorporates new model architectures and hardware backends.
For Kernel Developers & Contributors:
Learn exactly how to contribute effectively to vLLM. The session maps out where new logic should live, how the Python API bridges to custom kernels, and common pitfalls—such as custom ops not being invoked without proper registration. Debugging workflows (e.g., tracing CPU overhead) and benchmarking practices are covered to help contributors turn kernel prototypes into production-ready pull requests.
Whether you're optimizing production deployments or preparing to write the next major vLLM feature, this session will show you what makes vLLM tick—and how you can tap into its architecture to push LLM inference even further.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
At Ray Summit 2025, Tun Jian Tan from Embedded LLM shares an inside look at what gives vLLM its industry-leading speed, flexibility, and extensibility—offering a deep architectural walkthrough designed for everyone from day-to-day users to future kernel-level contributors.
The session uses a real-world case study—integrating AITER (AI Tensor Engine for ROCm) kernels and Multi-head Latent Attention (MLA) to accelerate DeepSeek-R1 inference on AMD GPUs—to illuminate how vLLM’s internals work and how new hardware and models plug into the system.
The talk is structured to deliver value across experience levels:
For vLLM Users & MLOps Engineers:
Gain a strong mental model of vLLM’s core architecture. Tun Jian explores internal components such as CustomOps, attention mechanisms, and the integration points for high-performance kernels—showing how vLLM achieves high throughput and rapidly incorporates new model architectures and hardware backends.
For Kernel Developers & Contributors:
Learn exactly how to contribute effectively to vLLM. The session maps out where new logic should live, how the Python API bridges to custom kernels, and common pitfalls—such as custom ops not being invoked without proper registration. Debugging workflows (e.g., tracing CPU overhead) and benchmarking practices are covered to help contributors turn kernel prototypes into production-ready pull requests.
Whether you're optimizing production deployments or preparing to write the next major vLLM feature, this session will show you what makes vLLM tick—and how you can tap into its architecture to push LLM inference even further.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com










