Uploaded May 2026 | Updated September 2026, 2 weeks ago

---

KV-cache quantization in MLX: I benchmarked TurboQuant on Apple Silicon to find out how much memory does it really save, and where it breaks.

⭐️⭐️⭐️ Thanks to VEED for sponsoring. Try the VEED Subtitles API on Fal: bit.ly/4fCVp4g ⭐️⭐️⭐️

The KV cache grows linearly with context and quickly becomes the memory wall on a Mac. 8-bit quantization is the easy fix, but 4-bit breaks greedy decode — because the two halves of the cache, K and V, behave completely differently: K feeds softmax (fragile), V gets averaged (forgiving). TurboQuant (Hadamard rotation + a Lloyd-Max codebook) is built to exploit that. I measured it across three architectures and four cache backends.

Measured (resident KV cache, Qwen 2.5 32B @ 32K):
• baseline fp16 — 7.85 GB
• scalar K8V4 — 3.17 GB (−60%)
• TurboQuant K4V4 — 2.07 GB (−74%)
• TurboQuant K3V3 — 1.71 GB (−78%)

…and the honest part: on Gemma 4 (hybrid sliding-window) every TurboQuant config tested
produces garbage in Python MLX, while scalar K8V4 stays coherent. The gain is real, but
it depends on your architecture and context length — you have to know which regime you're in.

Code and slides: github.com/juliensimon/turboquant-apple-metal

More content on Substack at airealist.ai