Your GPU Cluster Has 268 Settings. NVIDIA Put Them Under Version Control.

Installing a GPU cluster is the easy half. The hard half starts on the day one component moves and you can’t tell which combination was working last week.

NVIDIA AI Cluster Runtime, or AICR, is an attempt to make that combination a file. It snapshots a cluster, generates a recipe for the workload and platform, renders it into Helm values and manifests, deploys it, and validates the result before the application reaches the node pool.

The recipe is a version number for sixteen moving parts

A Dynamo recipe is not a single chart. It is a set. Alongside the Dynamo platform chart at 1.4.2, the recipe names the GPU Operator, Node Feature Discovery, the DRA driver for NVIDIA GPUs, KAI Scheduler, Grove, NVSentinel and Nodewright, plus cert-manager, the EBS CSI driver, a Prometheus stack and a gateway.

The recipe records what the team tested together. Pinned versions, compatibility constraints, node selectors, tolerations, checksums and the checks the cluster is meant to pass. Every entry there can move on its own release schedule.

The layering is the interesting part. Base layers set universal defaults, intent layers tune for training or inference, and hardware layers pin drivers and turn on accelerator features. Flip the intent and 5 components get swapped while 41 values change. Nobody should be surprised. The surprise is that the difference stayed invisible until someone wrote it down.

Dynamo 1.4 removed a dependency and kept two routing paths

In Dynamo 1.4, frontends and workers discover each other through Kubernetes resources, so the default recipe no longer installs NATS. It stays available as an opt-in, one less service to keep alive at three in the morning.

Two routing paths are supported. The first puts the Dynamo Frontend router behind a Kubernetes Service. It tokenizes the request, scores eligible workers on prefix overlap and current load, then dispatches over Dynamo’s TCP request plane. That’s the smallest topology.

The second hands ingress to the Gateway API Inference Extension and lets Dynamo’s endpoint picker choose the worker, with the gateway doing the proxying and each routable pod carrying a Dynamo sidecar. That path suits a team already running shared ingress and wants timeouts and fallbacks as gateway resources.

Both paths route well only if the router knows what each worker holds. This configuration uses worker-reported KV-cache events instead of predicting where a request landed. Prediction can’t see evictions, so the router’s view drifts from what the GPU actually contains, and routing degrades exactly when memory is under pressure. Dynamo 1.5 will swap the Go and CGO endpoint picker for a native Rust one, which removes a bridge from the request path.

The recipe proves the runtime, not your throughput

Validation covers expected cluster resources, a working nvidia-smi on accelerator nodes, GPU Operator health, DRA support, gang scheduling, autoscaling behavior, Prometheus metrics and gateway readiness. That is a stack audit, and a validated Kubernetes runtime is a claim about the stack rather than about the model. Performance validation runs separately because it consumes the GPUs under test.

The scope is narrow. Recipes target specific platforms and accelerators, with Amazon EKS on GB200 and GB300 NVL72 as the worked example, and NVIDIA owns every component it validates. A vendor writing the conformance test for its own stack is not a neutral party, and a green check is not a portability promise.

None of that makes it useless. The alternative is a wiki page from 18 months ago and one engineer who remembers which driver broke the fabric. A pinned recipe beats institutional memory on the day that engineer takes a new job.

Three questions for your own cluster this week. Which component versions are written down, and where? If a node came back from the vendor tomorrow, could you rebuild the exact stack that served traffic last month? And when the recipe moves, who decides whether you move with it?

Related reading. llm-d Hit 1.0 and the First Feature Is a Fair Queue. Mistral Opened a Trillion Parameter Model. The KV Cache Decided the Bill. Your 2027 Cluster Has a Host CPU Problem.