Is a Kubernetes Built for Containers Ready for AI?

Kubernetes handles containers. That is the problem.

Kubernetes was designed around a simple bet. Workloads are disposable, stateless, and cheap to move. The scheduler treats every pod as another unit of work on a homogeneous pool of nodes. CPU and memory are the resources that matter.

AI workloads break that bet on every axis. A training job is not disposable. It runs for days, holds a massive amount of state, and dies badly when interrupted. A single GPU node can cost more than the whole rest of the cluster. The scheduling philosophy, which assumes a pod can be killed and restarted anywhere, collides with the economics of a GPU that costs thousands of dollars an hour.

Teams discover this the hard way. The platform built to run containers well is being asked to run something that behaves like a different workload entirely.

GPU scheduling is the first real gap

The default scheduler does not understand a GPU at all. It knows that a pod declares a resource request, and it finds a node with enough of it. That is where the intelligence stops. It does not account for what a GPU actually is, a scarce, heterogeneous device that behaves differently depending on the model and the driver.

A modern GPU is a bundle of characteristics. Memory capacity, compute ability, the capacity to run mixed half-precision training, the interconnect and bandwidth between cards. Two GPUs that look identical on paper can be wildly different in practice. One might be ideal for a large language model inference job. The other might be a poor fit because its memory or its fabric does not match.

The NVIDIA GPU Operator exists to close that gap. It installs the drivers, the container toolkit, and the device plugin that exposes GPUs to the scheduler. It is now critical infrastructure for anyone running AI on Kubernetes seriously. But it is infrastructure, not scheduling intelligence. It gets the GPU visible to the cluster. It does not make the cluster smart about which GPU is right for which job.

Predictive autoscaling is where the savings live

Scaling is a bigger problem than placement. Traditional Kubernetes autoscaling is reactive. The Horizontal Pod Autoscaler watches CPU or memory, sees a spike, and adds replicas. It reacts after the fact. That works for a web server, where traffic is predictable and adding a pod is cheap. It does not work for a GPU.

GPU scaling is slow and expensive. Spinning up a GPU node can take minutes. A GPU has queueing latency, cold starts, and no cheap idle mode. If the cluster waits for demand to spike before adding GPUs, the user has already lost time waiting for the model to respond. Reactive scaling on GPUs produces a bad experience and a big bill.

That is why predictive autoscaling is emerging as the best practice. Instead of reacting to a spike, the system reads the history and anticipates it. It watches the pattern of inference requests, queue lengths, model serving load, and scheduled training jobs. It adds capacity before the spike lands. The goal is not just to keep up. The goal is to have the GPU waiting when the request arrives.

Tools like KEDA sit at the center of this. They scale on events and metrics, not just CPU. Model serving frameworks like vLLM and KServe expose the metrics that actually matter here, token throughput, queue depth, request latency, and in-flight requests. Those are the signals a predictive system can read. The platform learns that it needs five GPUs before the workday peak, and that demand on a Tuesday runs higher than on a Sunday.

The flip side of autoscaling is waste. Predictive scaling does not help if a GPU sits idle between jobs. The most expensive thing in an AI platform is a running GPU with nothing to do. In a normal infra world, idle capacity is a minor inefficiency. In an AI world, it is the biggest line item on the bill.

Good platforms solve this on both ends. They scale up predictively, then scale down aggressively the moment demand falls. They batch and queue work so a fraction of the GPU fleet stays busy instead of the whole fleet idling. They schedule training jobs around inference peaks. The platform team is no longer just placing pods. It is playing a continuous capacity game with money on the table.

The controls are getting better. But the underlying truth is that Kubernetes, at its core, was built around a model of cheap, interchangeable resources. A GPU fleet is neither cheap nor interchangeable. Until the scheduler and the autoscaler treat it that way, running AI on Kubernetes means compensating for the platform assumptions rather than leaning on them. The tools are close. The platform is not there yet.

The AI Infrastructure report walks the four layers under an enterprise AI platform, including why scheduling accelerators is a different job from scheduling web services.

Related reading. Kubernetes Was Built to Schedule Pods. GPUs Broke the Model.. The Migration Wave and the AI Shift Are the Same Decision. The Next Big AI Win Is Cutting Power Per Workload, Not Adding More Chips. AI Infrastructure Runs on Four Layers. Most Break Below the Model..