Uploaded July 2026 | Updated September 2026, 1 day ago
Managing AI model inference at scale is a nightmare — GPUs are scarce, scattered across clouds and regions, and the big models everyone actually cares about won't even fit on a single machine. This video introduces Modelplane, an early-stage open-source project that treats an entire spread of Kubernetes clusters as one unified inference fleet. The core insight is a clean separation between two roles that most platforms blur together: the platform side defines hardware classes and registers clusters, while developers simply declare what their model needs and let a scheduler handle placement across the fleet.
The demo walks through the full picture end to end — defining reusable GPU hardware classes, provisioning three real EKS clusters across three regions, deploying a vLLM-served model with a simple CEL-based hardware selector, and reaching all of its replicas through a single OpenAI-compatible endpoint on the control plane. The scheduler matches developer requirements against platform-published hardware, places replicas only where the GPUs actually fit, and routes every request through one front door. It's a v1alpha1 project, rough around the edges, but the architecture — one control plane, hardware-aware scheduling, and a single entry point for a geographically distributed fleet — tackles a genuinely hard problem in a clean and compelling way.
#KubernetesInference #GPUFleetManagement #MLOps
Consider joining the channel: youtube.com/c/devopstoolkit/join
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Transcript and commands: https://devopstoolkit.live/infrastructure-as-code/one-control-plane-for-every-gpu-cluster-modeplane
🔗 Modelplane: github.com/modelplaneai/modelplane
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ BlueSky: https://vfarcic.bsky.social
➡ LinkedIn: linkedin.com/in/viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: devopsparadox.com
💬 Live streams: youtube.com/c/DevOpsParadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Running Inference at Scale with Modelplane
03:15 Building a Kubernetes GPU Fleet
10:45 Deploying an LLM
16:28 Is Modelplane Worth It?
Managing AI model inference at scale is a nightmare — GPUs are scarce, scattered across clouds and regions, and the big models everyone actually cares about won't even fit on a single machine. This video introduces Modelplane, an early-stage open-source project that treats an entire spread of Kubernetes clusters as one unified inference fleet. The core insight is a clean separation between two roles that most platforms blur together: the platform side defines hardware classes and registers clusters, while developers simply declare what their model needs and let a scheduler handle placement across the fleet.
The demo walks through the full picture end to end — defining reusable GPU hardware classes, provisioning three real EKS clusters across three regions, deploying a vLLM-served model with a simple CEL-based hardware selector, and reaching all of its replicas through a single OpenAI-compatible endpoint on the control plane. The scheduler matches developer requirements against platform-published hardware, places replicas only where the GPUs actually fit, and routes every request through one front door. It's a v1alpha1 project, rough around the edges, but the architecture — one control plane, hardware-aware scheduling, and a single entry point for a geographically distributed fleet — tackles a genuinely hard problem in a clean and compelling way.
#KubernetesInference #GPUFleetManagement #MLOps
Consider joining the channel: youtube.com/c/devopstoolkit/join
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Transcript and commands: https://devopstoolkit.live/infrastructure-as-code/one-control-plane-for-every-gpu-cluster-modeplane
🔗 Modelplane: github.com/modelplaneai/modelplane
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please visit https://devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ BlueSky: https://vfarcic.bsky.social
➡ LinkedIn: linkedin.com/in/viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: devopsparadox.com
💬 Live streams: youtube.com/c/DevOpsParadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Running Inference at Scale with Modelplane
03:15 Building a Kubernetes GPU Fleet
10:45 Deploying an LLM
16:28 Is Modelplane Worth It?
