Uploaded May 2026 | Updated September 2026, 3 weeks ago
Is your LLM "production-ready," or is it just hiding a massive queue problem? In this video, I demonstrate how a single-user baseline can give you false confidence and how to use NVIDIA AIPerf to reveal the truth.
qainsights.com/99-of-requests-failed-and-my-dashboard-showed-green
What you’ll learn:
The Setup: Running Granite4:350m locally via Ollama.
The Trap: Why a single-user baseline (Concurrency 1) is a "baseline that lies."
The Reality Check: What happens to TTFT (Time to First Token) when you jump to 50 concurrent users.
The "Goodput" Metric: Using the --goodput flag to measure how many requests actually meet your SLO (and why 99% might be failing).
The Hidden Insight: Why ITL (Inter-Token Latency) staying solid means you have a queue problem, not a model problem.
Tools Mentioned:
NVIDIA AIPerf: github.com/ai-dynamo/aiperf
Ollama: Local LLM inference server.
Model: IBM Granite 4 350M.
➡️ Subscribe to my blog qainsights.com
➡️ Join QAInsights Community at https://qain.si/community
➡️ Join QAInsights Academy at https://qain.si/academy
➡️ Buy me a tea 🍵 buymeacoffee.com/qainsights
➡️ Get Certified in CKAD https://qain.si/startk8s
➡️ Get Certified in CKA https://qain.si/cka
➡️ My preferred DNS is NextDNS https://qain.si/nextdns
➡️ Learn Linux https://qain.si/linux
➡️ Get performance testing jobs real quick using Indeed → goo.gl/XAfCcE
➡️ Hostinger Web Hosting → goo.gl/MfwDyU
➡️ My Productivity Tools → goo.gl/2DfC5d
➡️ App Sumo for your business → goo.gl/zj92SA
➡️ Amazon → amzn.to/2L0Jv2n
➡️ TubeBuddy → tubebuddy.com/qainsights
➡️ LoadRunner Playlist youtube.com/playlist?list=PLJ9A48W0kpRIiVf8W7jMvf6Ao-naX3Ari
➡️ My first Udemy course entitled `Performance Testing using DevWeb` has been published https://qain.si/devweb
Is your LLM "production-ready," or is it just hiding a massive queue problem? In this video, I demonstrate how a single-user baseline can give you false confidence and how to use NVIDIA AIPerf to reveal the truth.
qainsights.com/99-of-requests-failed-and-my-dashboard-showed-green
What you’ll learn:
The Setup: Running Granite4:350m locally via Ollama.
The Trap: Why a single-user baseline (Concurrency 1) is a "baseline that lies."
The Reality Check: What happens to TTFT (Time to First Token) when you jump to 50 concurrent users.
The "Goodput" Metric: Using the --goodput flag to measure how many requests actually meet your SLO (and why 99% might be failing).
The Hidden Insight: Why ITL (Inter-Token Latency) staying solid means you have a queue problem, not a model problem.
Tools Mentioned:
NVIDIA AIPerf: github.com/ai-dynamo/aiperf
Ollama: Local LLM inference server.
Model: IBM Granite 4 350M.
➡️ Subscribe to my blog qainsights.com
➡️ Join QAInsights Community at https://qain.si/community
➡️ Join QAInsights Academy at https://qain.si/academy
➡️ Buy me a tea 🍵 buymeacoffee.com/qainsights
➡️ Get Certified in CKAD https://qain.si/startk8s
➡️ Get Certified in CKA https://qain.si/cka
➡️ My preferred DNS is NextDNS https://qain.si/nextdns
➡️ Learn Linux https://qain.si/linux
➡️ Get performance testing jobs real quick using Indeed → goo.gl/XAfCcE
➡️ Hostinger Web Hosting → goo.gl/MfwDyU
➡️ My Productivity Tools → goo.gl/2DfC5d
➡️ App Sumo for your business → goo.gl/zj92SA
➡️ Amazon → amzn.to/2L0Jv2n
➡️ TubeBuddy → tubebuddy.com/qainsights
➡️ LoadRunner Playlist youtube.com/playlist?list=PLJ9A48W0kpRIiVf8W7jMvf6Ao-naX3Ari
➡️ My first Udemy course entitled `Performance Testing using DevWeb` has been published https://qain.si/devweb
