Uploaded November 2026 | Updated November 2026, 3 hours ago
Which GPU an LLM request lands on decides time to first token and how much useful work you get from expensive hardware. This demo compares three routers in front of the same NVIDIA Dynamo 1.5 worker pool: HAProxy, an Istio gateway with the NVIDIA Dynamo endpoint picker, and F5 BIG-IP Next for Kubernetes (BNK) 2.4.
What the video covers:
- How an LLM request works: prefill builds the key value (KV) cache, then decode writes the answer token by token. Time to first token is the wait until prefill finishes.
- Why prefix reuse matters: enterprise apps resend long system prompts, RAG documents and chat history, so a worker that already holds the prefix in its KV cache can skip most of the prefill.
- How BNK scores each worker per request: cached prefix, queue, running requests, KV cache usage and expected time to first token.
- The lab setup: four workers (two fast, one medium, one slow) publishing KV events, with the same traffic sent through each router in turn.
- Scenario 1, mixed fleet at 60% capacity with 4000 token shared prompts: the slow worker's queue peaks at 102 requests with HAProxy, 26 with Istio and Dynamo, and 5 with BNK. Median time to first token is 237, 145 and 106 ms. The slowest 5% of requests wait 48 s, 4.1 s and 0.36 s. BNK served 84% of prompt tokens from the KV cache.
- Scenario 2, four identical workers at full load: the slowest 5% wait 7.1 s with HAProxy, 2.4 s with Istio and Dynamo, and 0.72 s with BNK 2.4, with 88% of prompt tokens found in the cache.
Lab note: no real hardware is used. NVIDIA BlueField DPUs are emulated with ordinary CPUs, the PCI Express link between DPU and GPU host is emulated over the network, and the GPU workers are simulated with Dynamo's mocker timed like real GPUs. BNK still picks a worker in about 2 milliseconds. In production, BNK runs on an NVIDIA BlueField DPU at the entrance to each GPU server, which is also where you can secure AI traffic and keep tenants apart.
Topics: LLM inference, KV cache, prefix caching, time to first token, tail latency, GPU utilization, AI traffic management, F5 BIG-IP Next for Kubernetes, NVIDIA Dynamo, NVIDIA BlueField, DPUs, Istio, HAProxy, Kubernetes.
▬▬▬ ABOUT DEVCENTRAL ▬▬▬
We're DevCentral: the how of F5, since 2003. A community of engineers, architects, and IT folks pushing each other forward, one hard-won answer at a time. We dig in, tackle the hard stuff, figure out what actually works, and share what we learn.
Keep going:
Forums, articles, CodeShare: community.f5.com
LinkedIn: linkedin.com/showcase/f5-devcentral
X: https://x.com/devcentral
The DevCentral advocates behind the videos:
Alessandro Fael Garcia → linkedin.com/in/alessfg · github.com/alessfg
Buu Lam → linkedin.com/in/buulam · https://x.com/buulam
Chase Abbott → linkedin.com/in/chaseabbott1
Jason Rahm → linkedin.com/in/jrahm · https://x.com/jasonrahm
Marko Sluga → linkedin.com/in/markosluga
Scott McAllister → linkedin.com/in/stmcallister
Which GPU an LLM request lands on decides time to first token and how much useful work you get from expensive hardware. This demo compares three routers in front of the same NVIDIA Dynamo 1.5 worker pool: HAProxy, an Istio gateway with the NVIDIA Dynamo endpoint picker, and F5 BIG-IP Next for Kubernetes (BNK) 2.4.
What the video covers:
- How an LLM request works: prefill builds the key value (KV) cache, then decode writes the answer token by token. Time to first token is the wait until prefill finishes.
- Why prefix reuse matters: enterprise apps resend long system prompts, RAG documents and chat history, so a worker that already holds the prefix in its KV cache can skip most of the prefill.
- How BNK scores each worker per request: cached prefix, queue, running requests, KV cache usage and expected time to first token.
- The lab setup: four workers (two fast, one medium, one slow) publishing KV events, with the same traffic sent through each router in turn.
- Scenario 1, mixed fleet at 60% capacity with 4000 token shared prompts: the slow worker's queue peaks at 102 requests with HAProxy, 26 with Istio and Dynamo, and 5 with BNK. Median time to first token is 237, 145 and 106 ms. The slowest 5% of requests wait 48 s, 4.1 s and 0.36 s. BNK served 84% of prompt tokens from the KV cache.
- Scenario 2, four identical workers at full load: the slowest 5% wait 7.1 s with HAProxy, 2.4 s with Istio and Dynamo, and 0.72 s with BNK 2.4, with 88% of prompt tokens found in the cache.
Lab note: no real hardware is used. NVIDIA BlueField DPUs are emulated with ordinary CPUs, the PCI Express link between DPU and GPU host is emulated over the network, and the GPU workers are simulated with Dynamo's mocker timed like real GPUs. BNK still picks a worker in about 2 milliseconds. In production, BNK runs on an NVIDIA BlueField DPU at the entrance to each GPU server, which is also where you can secure AI traffic and keep tenants apart.
Topics: LLM inference, KV cache, prefix caching, time to first token, tail latency, GPU utilization, AI traffic management, F5 BIG-IP Next for Kubernetes, NVIDIA Dynamo, NVIDIA BlueField, DPUs, Istio, HAProxy, Kubernetes.
▬▬▬ ABOUT DEVCENTRAL ▬▬▬
We're DevCentral: the how of F5, since 2003. A community of engineers, architects, and IT folks pushing each other forward, one hard-won answer at a time. We dig in, tackle the hard stuff, figure out what actually works, and share what we learn.
Keep going:
Forums, articles, CodeShare: community.f5.com
LinkedIn: linkedin.com/showcase/f5-devcentral
X: https://x.com/devcentral
The DevCentral advocates behind the videos:
Alessandro Fael Garcia → linkedin.com/in/alessfg · github.com/alessfg
Buu Lam → linkedin.com/in/buulam · https://x.com/buulam
Chase Abbott → linkedin.com/in/chaseabbott1
Jason Rahm → linkedin.com/in/jrahm · https://x.com/jasonrahm
Marko Sluga → linkedin.com/in/markosluga
Scott McAllister → linkedin.com/in/stmcallister










