Executive Summary
AI pilots rarely die on model quality. They die on everything around the model. Budgets, not benchmarks, decide which pilots become products. Data gravity, inference cost, latency, and scarce accelerators sort the winners long before launch day.
Pilot latency looks fine because nothing else competes for the hardware. Production differs. Requests queue, and cold starts reload multi-gigabyte weights, which can take tens of seconds before the first token. The p50 never mattered. The p95 is what users feel. GPU procurement lead times stretch for months, so a service owns its capacity outright instead of borrowing a card. Data gravity bites next, since vector indexes go stale overnight and retrieval steps around access controls the pilot never respected. Evaluation needs a golden set and a baseline, or teams argue about vibes. One SLO forces three numbers, cost per request, latency per request, and capacity headroom. Cost per completed task beats cost per token, because a cheap model that needs three retries is not cheap. Instrument token counts, queue depth, and error rates from week one, then let the curves come from reality.
Most AI pilots do not die because the model was wrong. They die because nobody built the service around it. The demo proved a prompt could work. It never proved the system could run every day at four in the morning.
That gap shows up in budgets, not benchmarks. Data gravity, inference cost, latency, and scarce GPUs decide which pilots become products. Here is what actually breaks, and how to plan for it before you promise a launch date.
A demo runs warm, and production starts cold
Pilots get lucky. One borrowed GPU, weights already sitting in memory, a curated dataset, and a handful of friendly testers. Latency looks excellent because nothing competes for the hardware.
Production strips away every one of those advantages at once. Concurrent requests queue behind each other. Cold starts reload multi-gigabyte weights, which can take tens of seconds before the first token appears. Your p50 looked fine in the demo. The p95 is what your users actually feel.
Cold starts also change with the workload. Add one to a chat interface and users assume it is broken. Add one to a nightly batch job and nobody notices. Knowing which you are building decides how much infrastructure the pilot really needs.
GPU scarcity makes this worse. Accelerators get allocated, not summoned, and procurement lead times stretch for months. A pilot borrows a card from a research team. A service has to own its capacity outright, with headroom for peak load and failover.
Run Kubernetes as the control plane for the workload and the scheduling problem becomes explicit. Kubernetes is a CNCF graduated project, and its concepts documentation is the place to start sizing requests and limits honestly.
Data gravity and evaluation gaps surface after launch
Data gravity is the quiet killer. A pilot reads a small, clean snapshot. Production reads live data scattered across systems, with permissions, residency rules, and freshness windows that nobody modeled during the demo.
Vector indexes go stale overnight. Retrieval quietly steps around access controls that the pilot never had to respect. Evaluation is the second gap. A pilot ships with one happy-path test set. A service needs a golden set, regression checks, and a way to catch quality drift over months.
Evals also need a baseline. Without one, you cannot tell whether a prompt change, a model upgrade, or a retrieval tweak made things better or worse. Teams that skip the baseline end up arguing about vibes.
Ownership is the third gap. A pilot has a champion. A service needs an on-call rotation, runbooks, and someone accountable when retrieval returns the wrong answer at scale.
Serving is where this gets concrete. KServe is a CNCF incubating project that standardizes inference on Kubernetes, and its introduction covers canary rollout and autoscaling. Kubeflow, now a CNCF graduated project, handles the pipeline side.
An SLO turns a demo into a service with budgets
An SLO forces three numbers you cannot dodge. Cost per request, latency per request, and capacity headroom. Inference cost compounds at volume. A token price that looks trivial in a pilot becomes a line item the finance team questions by quarter two.
Cost per completed task beats cost per token as a real metric. A cheap model that needs three retries to finish a job is not cheap at all.
Latency is a budget, not a hope. Time to first token, tokens per second, and retrieval overhead all draw from the same allowance. Guardrails and re-ranking add real milliseconds. Blow the budget and users abandon the feature before you ever see the savings.
Instrument before you scale. Log token counts, queue depth, and error rates from the first week of traffic. Then your cost and latency curves come from reality instead of a slide.
Reserve capacity for the traffic you promised, not the traffic you demoed. The NVIDIA GPU Operator manages drivers and device plugins across a fleet. A queueing layer then keeps aggressive teams from starving each other. Kueue is the Kubernetes project built for that job, and it pairs well with fair-share quotas.
None of this is exotic. It is the same discipline that keeps any service alive, applied to a workload with expensive hardware and fuzzy correctness. That is why pilots die in production, and why infrastructure teams usually end up saving them.
The AI Infrastructure report walks the four layers under an enterprise AI platform, including why scheduling accelerators is a different job from scheduling web services.
Related reading. Kubernetes for AI Is a Different Job Than Kubernetes for Apps. Agentic Resource Discovery Wants to Give Agents Their Own DNS. Is a Kubernetes Built for Containers Ready for AI?. AI Infrastructure Runs on Four Layers. Most Break Below the Model..
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data centre economics. No vendor spin.