Executive Summary
AI capacity planning is shifting from raw watts to work per watt. Every major accelerator vendor now sells on compute per watt, and those gains are real. They are also smaller than the demand curve. Power per node keeps climbing even as each unit of work gets cheaper, so a data center can draw more while spending less per task.
Scheduling holds the cheap wins. An idle accelerator still draws roughly a third of its rated power, so packing pods onto fewer busy GPUs cuts the bill without changing the workload. Training jobs can pause. Inference cannot, which pushes batch and training work into the gaps around the serving fleet. Quantized and smaller models cut memory and compute per token. Continuous batching groups requests that share model weights so one forward pass answers hundreds of them. The meter moves on all of these levers, and the teams pulling hardest are the ones that treat power as a scheduling input.
For years the answer to more AI was more hardware. Buy bigger chips, rent more GPUs, stand up more nodes. That answer is starting to fail, because the money and the power are not there. The new path is to do more work with the silicon you already own, and the industry is getting good at it.
The most interesting thing happening in AI infrastructure right now is not a new chip. It is the growing skill at squeezing more useful compute out of every watt of power.
The new chips save power per compute, not total power
Every major accelerator vendor talks about the same number now. Compute per watt. The density of transistors has made it possible to train and serve models with far less energy than anyone thought a few years ago. In practice, a single accelerator can handle a much larger chunk of a model, and racks can shrink without losing throughput.
Here is the catch. The demand for AI is growing faster than those efficiency gains. So absolute power per accelerator keeps climbing. A new generation of chips still raises the power ceiling per node, even as it lowers the power per unit of work. The result is a data center that draws more total power even as each part works more efficiently. Efficiency is not the same thing as frugality.
Scheduling is where the real savings are
A lot of the cheapest wins do not require new silicon at all. They require the scheduler to be smarter about the work it is placing. Power efficiency at the rack level is increasingly a software problem, not a hardware problem.
The goal is to get more throughput per accelerator, and there are several well-understood levers. Packing workloads densely so a GPU is never sitting idle is the most obvious. An idle accelerator still draws power, often a third of its rated draw, which is pure waste. Packing many pods onto fewer, fully used accelerators cuts the number of running nodes and lowers the power bill without changing the workload.
There is also the question of when work runs. Training runs can often pause. Inference cannot. So the scheduling decision is to right-size the always-on serving fleet for the peaks, then push batch and training into the gaps. Some operators use spot capacity or transient instances for the parts of a job that can be rescheduled, which lets them use machines that are otherwise sitting dark.
The model itself is another lever. A model that is smaller or quantized needs less memory and less compute per token. Techniques that shrink the working set, cut redundant computation, and stop the generation stage from recomputing context all move the meter. Across a large fleet these add up to a materially lower energy bill per request. Continuous batching is one of those quiet wins. Instead of paying the full compute cost for one request at a time, the serving layer groups many requests that share model weights, so a single forward pass answers hundreds of them. The same hardware that used to serve a trickle now handles a steady stream, and the power per request falls sharply.
The goal is work per watt, not raw watts
The industry is converging on a single useful metric, and it is not the size of the build. It is how much value comes out of each watt. The operators moving fastest are not the ones buying the biggest chips. They are the ones with the most disciplined scheduling and the tightest match between model size, hardware, and the workload.
This is why the conversation about AI power is more complicated than it looks from the outside. Yes, AI consumes a lot of electricity. But the marginal cost of a unit of work is falling. That is the part that matters for whoever pays the operating bill. The data centers that win are the ones that treat power as a first-class scheduling input and not a line item they pay for after the fact. Learn more about NIST.
Data center power is the other half of this story. The 2026 State of Enterprise Infrastructure report covers where the megawatts are going and why grid access gates new capacity.
Related reading. The Cost of AI Is Finally Falling. The Cost of Using It Is Not.. Kubernetes for AI Is a Different Job Than Kubernetes for Apps. Your Kubernetes Bill Is an Architecture Problem, Not a FinOps Problem. AI Infrastructure Runs on Four Layers. Most Break Below the Model..
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.