Executive Summary
GPUs cost too much to treat as whole numbers. Kubernetes schedules counts, not devices, and that mismatch wastes the scarcest hardware in AI. Buying accelerators was never the hard part. Deciding which workload gets one, and when, is where clusters lose money.
Device plugins went stable in Kubernetes 1.26 and still advertise a count. Ask for one GPU and the scheduler hands over a whole card. Gang scheduling is the first gap. A training job with eight workers and seven scheduled leaves seven GPUs idle while it waits. Topology is the second. Two cards on the same NVLink domain move data far faster than two across PCIe, and the default scheduler is blind to that. Multi-Instance GPU splits a card into isolated slices with dedicated memory, though a slice with too little memory fails the model outright. Kueue adds quotas, priority, and fair sharing. Dynamic Resource Allocation is the longer fix and still early. A 2026 study measured production clusters at roughly 5 percent utilization, and the best-tuned fleet barely reached 50 percent. Read vendor research with care, then check DCGM metrics for framebuffer pressure.
GPUs are the scarcest resource in AI infrastructure, and the hardest part is not buying them. It is deciding which workload gets one and when. Kubernetes was built to place pods. Scheduling accelerators was never the design goal, and that gap is now the central problem.
Most clusters treat a GPU as a single integer. A pod asks for one, and the scheduler hands over a whole card. That model is close to a decade old, and it wastes hardware. A modern accelerator has memory, topology, partition modes, and health state. Kubernetes sees almost none of it. Device plugins went stable in Kubernetes 1.26, and they still advertise a count, not a device.
Kubernetes Schedules Counts, and Accelerators Are Not Counts
The NVIDIA GPU Operator handles the plumbing. It installs drivers, exposes the device plugin, and labels nodes. It works well. It also inherits the same ceiling, because the resource it publishes is nvidia.com/gpu, a plain number.
Real scheduling needs more. Distributed training needs all eight pods at once or none. Practitioners call that gang scheduling, and the default scheduler does not do it. Gang scheduling sounds like a niche feature until you run it. A training job with eight workers and only seven scheduled leaves seven GPUs idle while it waits. That is a rounding error on one job. Across a fleet, it is real money.
A low-priority batch job can also hold every GPU on a node while a serving workload waits. Priority and preemption help, but only if you configure them. Most teams never do. The deeper issue is that the scheduler assumes resources are fungible. CPUs are interchangeable. GPUs are not. One H100 differs from another once memory and interconnect enter the picture.
Topology matters too. Two cards on the same NVLink domain move data far faster than two across PCIe. Kubernetes is blind to that. So the scheduler places a pod that then stalls on a slow link, burning the exact resource you meant to protect.
Sharing the Hardware Is the Fastest Win You Have
You do not need new silicon to raise utilization. You need to share what you own. Multi-Instance GPU splits one card into isolated slices with dedicated memory. That fits small models and many inference replicas. Time slicing and MPS do something coarser, letting several pods take turns on one device. The catch is memory. A slice with too little of it fails the model outright, so the profile has to match the workload.
Inference and training want opposite things. Training is greedy, long, and wants the whole card. Inference is bursty and often needs a fraction. Running both on fixed hardware without sharing guarantees idle capacity somewhere. Partitioning fixes that. It also forces a real decision about who deserves a whole GPU. That decision is policy, not plumbing, and platform teams should own it.
Dynamic Resource Allocation is the longer fix. It lets a pod claim a device by attributes instead of a count. That is a genuine change to how scheduling works under the hood. It is also still early, so treat it as the direction of travel, not a switch you flip this quarter.
Kueue fills the queueing gap. It is a CNCF sandbox project, and it adds quotas, priority, and fair sharing across teams. Kubernetes itself is a graduated CNCF project, which is what lets a subproject build on a stable base.
Nobody Actually Knows Their Real GPU Utilization
Vendor telemetry suggests the average sits far lower than most operators assume. One 2026 study measured production clusters at roughly 5 percent. The best-tuned fleet in that same analysis barely reached 50 percent. Even allowing for error, the gap is large. Read vendor research with care, since the authors sell optimization software.
Start with DCGM metrics. Compute utilization, framebuffer use, and power draw show what the cards actually do, not what the cluster thinks they do. Then hunt the idle pattern. Nightly batch jobs that reserve cards all day. Training runs that finish and leave nodes warm. Those gaps are where better scheduling pays for itself. Set alerts on framebuffer pressure too, because a card can look busy while it is really just waiting.
The lesson is plain. GPUs cost too much to treat as integers. Get the scheduling model right, and the same hardware carries far more work.
The AI Infrastructure report walks the four layers under an enterprise AI platform, including why scheduling accelerators is a different job from scheduling web services.
Related reading. Kubernetes for AI Is a Different Job Than Kubernetes for Apps. Is a Kubernetes Built for Containers Ready for AI?. The Next Big AI Win Is Cutting Power Per Workload, Not Adding More Chips. AI Infrastructure Runs on Four Layers. Most Break Below the Model..
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data centre economics. No vendor spin.