KV Cache and Long-Context Inference: Solutions for Inference at Scale | VAST Data | Ray Summit 2026 @anyscale
KV Cache and Long-Context Inference: Solutions for Inference at Scale | VAST Data | Ray Summit 2026  @anyscale
Uploaded September 2026 | Updated November 2026, 4 days ago
As agentic AI drives ever-longer context windows, KV cache has become the new memory wall, dwarfing GPU HBM capacity and forcing costly recomputation on every session resume.

At Ray Summit 2026, Vaughn Stewart, VP of Systems Engineering at VAST Data, covers how offloading KV cache to VAST's flash-based storage with NVIDIA Dynamo and BlueField-4/CMX architecture eliminates that tax: 20x faster time to first token, 90 percent GPU time savings, and up to 64 percent lower three-year TCO.

You'll leave with sizing guidance, benchmark data, and a blueprint for scaling inference context without scaling GPU spend.

Liked this video? Check out other Ray Summit breakout session recordings: youtube.com/playlist?list=PLZNaJyYZRPdo

Subscribe to our YouTube channel to stay up-to-date on the future of AI: youtube.com/c/anyscale

🔗 Connect with us:
LinkedIn: linkedin.com/company/joinanyscale
X: https://x.com/anyscalecompute
Website: anyscale.com
KV Cache and Long-Context Inference: Solutions for Inference at Scale | VAST Data | Ray Summit 2026Reward Hacking and the Gaps in Agentic AI | Ion Stoica (Anyscale) | Ray Summit 2026How AWS Scales Reinforcement Learning Across Thousands of GPUs | Ray Summit 2025Distributed Model Training with Ray at Capital One | Ray Summit 2025State of vLLM 2026 | Inferact | Ray Summit 2026From Benchmark #1 to Production | Nebius | Ray Summit 2026Foundation Model Distributed Training with Ray | Anyscale | Ray Summit 2026High-Throughput Inference for Synthetic Data & Evals at Sutro | Ray Summit 2025How a Transformers Model Loads in vLLM | Hugging Face | Ray Summit 2026NVIDIA’s Framework for Scalable Data Curation | Ray Summit 2025Koba: Disaggregated Data Loading for Multimodal Training at Luma AI | Luma AI | Ray Summit 2026A Compiler for Christmas | Brendan Burns #raysummit
Anyscale |

KV Cache and Long-Context Inference: Solutions for Inference at Scale | VAST Data | Ray Summit 2026

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER