Kubernetes for AI Is a Different Job Than Kubernetes for Apps

Most teams learn Kubernetes, the CNCF graduated orchestrator, by serving web apps. The habits that follow are wrong for AI. A stateless pod behind a Service, a CPU request, and a horizontal autoscaler works fine for a web tier. It collapses when the workload pins 80 GB of model weights to a card worth more than a rack of servers. Kubernetes for AI is a different problem, and the defaults mislead.

Start with the resource model. CPU and memory requests describe how much a container wants, and the kubelet packs pods together to share the rest. GPUs do not work that way. A pod asks for a whole device and gets exclusive use of it. Kubernetes GPU scheduling counts devices, it does not measure them. Ask for one GPU and you get one card, whether the job needs 5 percent of it or all of it. Teams that treat a GPU like a large CPU number end up with idle silicon and a scheduler that reports the node as full.

Device counts do not fragment. Eight accelerators on one node and a pod that requests two leaves room for three more, never four. Memory and CPU leave gaps the kubelet can fill. A device count does not, so half-used nodes become a steady state, not a bug.

Oversubscription is the usual answer. The NVIDIA GPU Operator splits a card with time slicing or MIG. Time slicing interleaves contexts and offers no memory isolation. MIG carves hard partitions with real isolation, but only on the newer data center cards and only in fixed slice shapes. Neither one repairs the scheduling story, because the scheduler still sees a device count.

Memory, Not Compute, Sets the Serving Ceiling

Inference has a shape that web apps do not. A model loads once and then answers requests for hours. Pulling a 70 billion parameter model from storage takes minutes. That load happens inside the pod, so a cold start is not measured in seconds. A vLLM process sizes its KV cache against the whole card when it starts. Restart it and you pay the load again, plus warmup. A readiness probe tuned for a Go binary will kill a pod that is still reading weights.

Autoscaling breaks on the same fault line. A horizontal pod autoscaler reacts to CPU pressure or request rate. What does a second replica cost here? Another whole GPU and another multi-minute load. Scale to zero and the next user waits for a full model load. The metrics mislead too. Request rate says nothing useful when a second replica takes minutes to become ready. Queue depth and time to first token track the truth.

Long-Running Jobs Are Not Stateless Pods

A Deployment assumes pods are interchangeable and disposable. A training run is neither. It holds state in GPU memory, it may span many nodes for weeks, and an eviction does not restart a small unit of work. It throws away everything since the last checkpoint. Teams that schedule training the way they schedule a web tier discover this when a routine node drain kills three days of progress.

Multi-node training needs every worker to start together. The default scheduler places pods one at a time and has no notion of a group. A partly scheduled job holds a GPU hostage while it waits for the rest, which is how a cluster that looks 60 percent utilized ends up refusing new work. Queueing layers such as Kueue add gang admission, quotas, and preemption so a job either runs whole or waits in line.

The Scheduler Was Built for CPUs

The default scheduler balances pods across nodes using CPU, memory, and taints. It does not reason about NVLink, PCIe topology, or which two accelerators share a switch. Place two pods on the wrong pair and a fast model runs slowly for no visible reason. Dynamic Resource Allocation is now stable, which finally gives the scheduler a way to express device attributes and topology instead of a bare integer. It helps. It is not the default.

Serving frameworks assumed the gap and papered over it. KServe, a CNCF incubating project, wraps routing, revision rollout, and scale to zero behind a predictor object. It hides the pod plumbing. Hiding is not removing. The GPU still gets requested, the model still gets loaded, and the node still has to fit.

So budget for whole devices, not fractions. Build node pools per accelerator type. Size your autoscaler around model load time, not request rate. Put a queue in front of the scheduler so gangs wait instead of half-starting. Treat a serving pod as a stateful, expensive, slow-to-start process, because that is what it is. The web app patterns are a bad teacher here.

The AI Infrastructure report walks the four layers under an enterprise AI platform, including why scheduling accelerators is a different job from scheduling web services.

Related reading. is Kubernetes built for containers ready for AI.