Uploaded March 2026 | Updated September 2026, 2 weeks ago
Two years after parts 1 (youtu.be/kw7S-3s50uk) and 2 (youtu.be/fXBBwCIA0Ds), the quantization landscape has changed completely. FP8 is the new default; the serving stack has absorbed quantization, and MoE models have broken the old assumptions. This video shows you what practitioners actually use today.
⭐️⭐️⭐️ More content on Substack at airealist.ai ⭐️⭐️⭐️
In Part 3 of this series, I'm showing you exactly what to run, on which GPU, with which tool, and what breaks when you get it wrong. One model, start to finish: Arcee Trinity Mini (26B MoE, 3B active, 128 experts).
What's covered:
→ Why FP8 is the new BF16 — essentially lossless, one flag, works everywhere
→ The three things people call "4-bit" (INT4 vs MXFP4 vs NVFP4) and why they're completely different
→ Serving with vLLM: BF16, FP8 on-the-fly, FP8 checkpoint, with verified H100 numbers
→ Why MoE models break standard quantization and how to fix it
→ Creating your own checkpoints with llm-compressor and NVIDIA ModelOpt
→ NVFP4 on Blackwell: real 4-bit tensor core math, what works and what doesn't yet
→ The FP4 deep dive: E2M1 bit layout, MXFP4 vs NVFP4 scale factors, worked error example
→ Decision tree: which quantization path for your hardware and use case
*** Slides
fr.slideshare.net/slideshow/advanced-quantization-techniques-for-large-language-models-in-2026-a4c8/286754686
*** Arcee Trinity Mini:
26B total parameters, 3B active per token, 128 experts, 128K context, Apache 2.0.
huggingface.co/arcee-ai/Trinity-Mini
NVFP4: huggingface.co/arcee-ai/Trinity-Mini-NVFP4
FP8: huggingface.co/arcee-ai/Trinity-Mini-FP8-Block
#llm #quantization #fp8 #fp4 #nvfp4 #vllm #inference #nvidia #blackwell #moe #arcee
Two years after parts 1 (youtu.be/kw7S-3s50uk) and 2 (youtu.be/fXBBwCIA0Ds), the quantization landscape has changed completely. FP8 is the new default; the serving stack has absorbed quantization, and MoE models have broken the old assumptions. This video shows you what practitioners actually use today.
⭐️⭐️⭐️ More content on Substack at airealist.ai ⭐️⭐️⭐️
In Part 3 of this series, I'm showing you exactly what to run, on which GPU, with which tool, and what breaks when you get it wrong. One model, start to finish: Arcee Trinity Mini (26B MoE, 3B active, 128 experts).
What's covered:
→ Why FP8 is the new BF16 — essentially lossless, one flag, works everywhere
→ The three things people call "4-bit" (INT4 vs MXFP4 vs NVFP4) and why they're completely different
→ Serving with vLLM: BF16, FP8 on-the-fly, FP8 checkpoint, with verified H100 numbers
→ Why MoE models break standard quantization and how to fix it
→ Creating your own checkpoints with llm-compressor and NVIDIA ModelOpt
→ NVFP4 on Blackwell: real 4-bit tensor core math, what works and what doesn't yet
→ The FP4 deep dive: E2M1 bit layout, MXFP4 vs NVFP4 scale factors, worked error example
→ Decision tree: which quantization path for your hardware and use case
*** Slides
fr.slideshare.net/slideshow/advanced-quantization-techniques-for-large-language-models-in-2026-a4c8/286754686
*** Arcee Trinity Mini:
26B total parameters, 3B active per token, 128 experts, 128K context, Apache 2.0.
huggingface.co/arcee-ai/Trinity-Mini
NVFP4: huggingface.co/arcee-ai/Trinity-Mini-NVFP4
FP8: huggingface.co/arcee-ai/Trinity-Mini-FP8-Block
#llm #quantization #fp8 #fp4 #nvfp4 #vllm #inference #nvidia #blackwell #moe #arcee
