Uploaded October 2026 | Updated November 2026, 3 hours ago
This lab demo shows how F5 BIG-IP Next for Kubernetes (BNK 2.4) routes LLM inference requests across a shared pool of GPU workers running NVIDIA Dynamo 1.5.
Why routing matters for LLM inference: each request runs a prefill phase, where the GPU reads the full prompt and builds its KV cache, followed by a decode phase that generates the answer token by token. Enterprise applications resend the same long prefix with every request, including system prompts, RAG documents, and chat history. A worker that already holds that prefix in KV cache skips most of the prefill work, so the router has to send each request to a worker that has the prefix cached while avoiding workers with long queues.
What the lab compares: the same traffic runs through three routers in turn: HAProxy (a classic reverse proxy with round robin, blind to cache and load), Istio with the NVIDIA Dynamo endpoint picker (which follows KV events and load), and F5 BIG-IP Next for Kubernetes. For each request, BNK scores every worker on cached prefix, queue depth, running requests, KV cache usage, and expected time to first token.
Scenario 1 (mixed fleet at 60% capacity, 4,000-token shared prompts): median time to first token was 237 ms (HAProxy), 145 ms (Istio with Dynamo), and 106 ms (BNK). For the slowest 5% of requests, time to first token was 48 seconds (HAProxy), 4.1 seconds (Istio with Dynamo), and 0.36 seconds (BNK). Cache hits covered 34%, 62%, and 84% of prompt tokens respectively. BNK sent the slow worker about 9% of requests, roughly half of what HAProxy sent, and kept its longest queue at 5 requests compared with 26 and 102.
Scenario 2 (four identical workers at full load): the slowest 5% waited 7.1 seconds with HAProxy, 2.4 seconds with Istio and Dynamo, and 0.72 seconds with BNK, which found the cached prefix for 88% of prompt tokens.
Lab notes: this is an emulated environment. The DPUs are simulated on CPUs, the PCIe link between DPU and GPU host runs over the network, and the GPU workers are simulated with Dynamo's mocker, timed to match real GPUs. These extra hops are not present in production, where BNK runs on an NVIDIA BlueField DPU at the entrance to each GPU server. Even so, BNK selected a worker in about 2 milliseconds.
Topics covered: LLM inference routing, KV cache, prefill and decode, time to first token (TTFT), prefix caching, NVIDIA Dynamo, Kubernetes, NVIDIA BlueField DPU, F5 BIG-IP Next for Kubernetes, GPU load balancing, AI traffic security and multi-tenancy.
▬▬▬ ABOUT DEVCENTRAL ▬▬▬
We're DevCentral: the how of F5, since 2003. A community of engineers, architects, and IT folks pushing each other forward, one hard-won answer at a time. We dig in, tackle the hard stuff, figure out what actually works, and share what we learn.
Keep going:
Forums, articles, CodeShare: community.f5.com
LinkedIn: linkedin.com/showcase/f5-devcentral
X: https://x.com/devcentral
The DevCentral advocates behind the videos:
Alessandro Fael Garcia → linkedin.com/in/alessfg · github.com/alessfg
Buu Lam → linkedin.com/in/buulam · https://x.com/buulam
Chase Abbott → linkedin.com/in/chaseabbott1
Jason Rahm → linkedin.com/in/jrahm · https://x.com/jasonrahm
Marko Sluga → linkedin.com/in/markosluga
Scott McAllister → linkedin.com/in/stmcallister
This lab demo shows how F5 BIG-IP Next for Kubernetes (BNK 2.4) routes LLM inference requests across a shared pool of GPU workers running NVIDIA Dynamo 1.5.
Why routing matters for LLM inference: each request runs a prefill phase, where the GPU reads the full prompt and builds its KV cache, followed by a decode phase that generates the answer token by token. Enterprise applications resend the same long prefix with every request, including system prompts, RAG documents, and chat history. A worker that already holds that prefix in KV cache skips most of the prefill work, so the router has to send each request to a worker that has the prefix cached while avoiding workers with long queues.
What the lab compares: the same traffic runs through three routers in turn: HAProxy (a classic reverse proxy with round robin, blind to cache and load), Istio with the NVIDIA Dynamo endpoint picker (which follows KV events and load), and F5 BIG-IP Next for Kubernetes. For each request, BNK scores every worker on cached prefix, queue depth, running requests, KV cache usage, and expected time to first token.
Scenario 1 (mixed fleet at 60% capacity, 4,000-token shared prompts): median time to first token was 237 ms (HAProxy), 145 ms (Istio with Dynamo), and 106 ms (BNK). For the slowest 5% of requests, time to first token was 48 seconds (HAProxy), 4.1 seconds (Istio with Dynamo), and 0.36 seconds (BNK). Cache hits covered 34%, 62%, and 84% of prompt tokens respectively. BNK sent the slow worker about 9% of requests, roughly half of what HAProxy sent, and kept its longest queue at 5 requests compared with 26 and 102.
Scenario 2 (four identical workers at full load): the slowest 5% waited 7.1 seconds with HAProxy, 2.4 seconds with Istio and Dynamo, and 0.72 seconds with BNK, which found the cached prefix for 88% of prompt tokens.
Lab notes: this is an emulated environment. The DPUs are simulated on CPUs, the PCIe link between DPU and GPU host runs over the network, and the GPU workers are simulated with Dynamo's mocker, timed to match real GPUs. These extra hops are not present in production, where BNK runs on an NVIDIA BlueField DPU at the entrance to each GPU server. Even so, BNK selected a worker in about 2 milliseconds.
Topics covered: LLM inference routing, KV cache, prefill and decode, time to first token (TTFT), prefix caching, NVIDIA Dynamo, Kubernetes, NVIDIA BlueField DPU, F5 BIG-IP Next for Kubernetes, GPU load balancing, AI traffic security and multi-tenancy.
▬▬▬ ABOUT DEVCENTRAL ▬▬▬
We're DevCentral: the how of F5, since 2003. A community of engineers, architects, and IT folks pushing each other forward, one hard-won answer at a time. We dig in, tackle the hard stuff, figure out what actually works, and share what we learn.
Keep going:
Forums, articles, CodeShare: community.f5.com
LinkedIn: linkedin.com/showcase/f5-devcentral
X: https://x.com/devcentral
The DevCentral advocates behind the videos:
Alessandro Fael Garcia → linkedin.com/in/alessfg · github.com/alessfg
Buu Lam → linkedin.com/in/buulam · https://x.com/buulam
Chase Abbott → linkedin.com/in/chaseabbott1
Jason Rahm → linkedin.com/in/jrahm · https://x.com/jasonrahm
Marko Sluga → linkedin.com/in/markosluga
Scott McAllister → linkedin.com/in/stmcallister










