Uploaded November 2025 | Updated September 2026, 2 weeks ago
At Ray Summit 2025, Manoj Krishnan and Brittany Rockwell from Google share an in-depth look at the new optimized TPU backend in vLLM, designed to unify and accelerate large-scale inference across both PyTorch and JAX models under a single, consolidated codepath.
They begin by highlighting how this new backend preserves vLLM’s hallmark ease-of-use and portability—allowing developers to move seamlessly between hardware types—while introducing a suite of next-generation TPU capabilities purpose-built for XL-scale model deployments. These include:
Disaggregated serving for more flexible resource allocation
Advanced parallelism strategies for Mixture-of-Experts (MoE) models
Highly optimized Pallas kernels for maximized TPU performance
Enhanced multimodal support tailored for large, heterogeneous model architectures
Manoj and Brittany walk through architectural details, performance optimizations, and practical deployment patterns that make the new TPU backend a powerful option for teams running frontier-scale models.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
At Ray Summit 2025, Manoj Krishnan and Brittany Rockwell from Google share an in-depth look at the new optimized TPU backend in vLLM, designed to unify and accelerate large-scale inference across both PyTorch and JAX models under a single, consolidated codepath.
They begin by highlighting how this new backend preserves vLLM’s hallmark ease-of-use and portability—allowing developers to move seamlessly between hardware types—while introducing a suite of next-generation TPU capabilities purpose-built for XL-scale model deployments. These include:
Disaggregated serving for more flexible resource allocation
Advanced parallelism strategies for Mixture-of-Experts (MoE) models
Highly optimized Pallas kernels for maximized TPU performance
Enhanced multimodal support tailored for large, heterogeneous model architectures
Manoj and Brittany walk through architectural details, performance optimizations, and practical deployment patterns that make the new TPU backend a powerful option for teams running frontier-scale models.
Subscribe to our YouTube channel to stay up-to-date on the future of AI! youtube.com/c/anyscale
🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
