Uploaded April 2026 | Updated September 2026, 2 days ago
Running AI inference workloads on Kubernetes presents unique challenges. Traditional load balancing methods designed for standard web APIs simply fall short when it comes to the nuanced resource management required for generative LLMs.
In this demo, we explore how the NGINX Gateway Fabric Inference Extension transforms your Kubernetes environment by acting as a dedicated Inference Gateway. Discover how to move beyond basic proxying by utilizing a powerful two-stage architecture—combining standard HTTPRoutes with the InferencePool CRD and Endpoint Picker. This allows NGINX to make intelligent, real-time routing decisions based on queue depth and GPU availability.
What you will see in this demo:
0:11 - The Challenges of AI Workloads: Why standard load balancing fails for GPUs.
0.51 - Two-Stage Orchestration: Understanding the InferencePool and Endpoint Picker.
2:24 - Model-Aware Routing: Inspecting traffic paths to offload lightweight tasks to CPUs and reserve premium GPUs for heavy text generation.
3:46 - Canary Testing: Validating new models safely using native 90/10 traffic splitting.
4:49 - Cost Optimization: Using custom HTTP headers to guarantee SLAs and reserve compute for mission-critical queries.
🔗 Helpful Links & Resources:
NGINX Gateway Fabric Documentation: [docs.nginx.com/nginx-gateway-fabric]
GitHub Repository / Lab Guides: [github.com/nginx/nginx-gateway-fabric]
Notes:
- Contributed by: Akash Ananthanarayanan
- Related Article: None
⬇️⬇️ JOIN THE COMMUNITY! ⬇️⬇️⬇️
DevCentral is an online community of technical peers dedicated to learning, exchanging ideas, and solving problems—together.
Find all our platform links ⬇️ and follow our Community Evangelists! 👋
➡️ DEVCENTRAL: community.f5.com
➡️ YOUTUBE: youtube.com/devcentral
➡️ LINKEDIN: linkedin.com/showcase/f5-devcentral
➡️ X: https://x.com/devcentral
Your Community Evangelists:
👋 Jason Rahm: linkedin.com/in/jrahm | https://x.com/jasonrahm
👋 Buu Lam: linkedin.com/in/buulam | https://x.com/buulam
👋 Chase Abbott: linkedin.com/in/chaseabbott1
Running AI inference workloads on Kubernetes presents unique challenges. Traditional load balancing methods designed for standard web APIs simply fall short when it comes to the nuanced resource management required for generative LLMs.
In this demo, we explore how the NGINX Gateway Fabric Inference Extension transforms your Kubernetes environment by acting as a dedicated Inference Gateway. Discover how to move beyond basic proxying by utilizing a powerful two-stage architecture—combining standard HTTPRoutes with the InferencePool CRD and Endpoint Picker. This allows NGINX to make intelligent, real-time routing decisions based on queue depth and GPU availability.
What you will see in this demo:
0:11 - The Challenges of AI Workloads: Why standard load balancing fails for GPUs.
0.51 - Two-Stage Orchestration: Understanding the InferencePool and Endpoint Picker.
2:24 - Model-Aware Routing: Inspecting traffic paths to offload lightweight tasks to CPUs and reserve premium GPUs for heavy text generation.
3:46 - Canary Testing: Validating new models safely using native 90/10 traffic splitting.
4:49 - Cost Optimization: Using custom HTTP headers to guarantee SLAs and reserve compute for mission-critical queries.
🔗 Helpful Links & Resources:
NGINX Gateway Fabric Documentation: [docs.nginx.com/nginx-gateway-fabric]
GitHub Repository / Lab Guides: [github.com/nginx/nginx-gateway-fabric]
Notes:
- Contributed by: Akash Ananthanarayanan
- Related Article: None
⬇️⬇️ JOIN THE COMMUNITY! ⬇️⬇️⬇️
DevCentral is an online community of technical peers dedicated to learning, exchanging ideas, and solving problems—together.
Find all our platform links ⬇️ and follow our Community Evangelists! 👋
➡️ DEVCENTRAL: community.f5.com
➡️ YOUTUBE: youtube.com/devcentral
➡️ LINKEDIN: linkedin.com/showcase/f5-devcentral
➡️ X: https://x.com/devcentral
Your Community Evangelists:
👋 Jason Rahm: linkedin.com/in/jrahm | https://x.com/jasonrahm
👋 Buu Lam: linkedin.com/in/buulam | https://x.com/buulam
👋 Chase Abbott: linkedin.com/in/chaseabbott1
