Executive Summary
Cloud Pathways is now generally available on Google Kubernetes Engine, and what went live is a scheduler, not silicon. A single controller process dispatches XLA work across thousands of TPU chips spread over multiple slices, and the compilation cache sits in Cloud Storage rather than on any one host. Google reports 130 million Cloud TPU chip-hours through Pathways in the last 90 days. One customer, described only as a large consumer technology company, accounts for more than 100 million chip-hours every quarter across tens of thousands of v5p, Trillium and Ironwood chips.
The verdict is that Google productized a DeepMind internal tool because at that density the submission layer becomes the cost center. Elastic training is the feature that matters most. A dropped slice now resizes the job to N-1 instead of killing the client. The limit is that Pathways speaks TPU and nothing else. GPU fleets get no benefit, and the Slurm path most research clusters actually submit through is still in preview.
Google’s distributed runtime for TPU fleets left preview on 9 November. Cloud Pathways is generally available on Google Kubernetes Engine for Cloud TPU v5e, v5p, Trillium (v6e) and Ironwood (TPU7x), shipped as versioned container images and requiring JAX 0.10.1 or later.
Understand the problem first. In ordinary multi-controller JAX, every TPU host runs its own Python interpreter with a local view of only its own devices, and each host executes the same SPMD program in lockstep. Two controllers means two views of one job. Getting coordinated work across slices turns into a negotiation between independent processes.
One process now owns the whole topology
Cloud Pathways replaces that with a single controller. A client sets JAX_PLATFORMS=proxy and then sees multiple large TPU slices as though the chips were wired to the same machine. The controller owns graph evaluation, dispatches XLA HLO to the slices asynchronously, and keeps that compilation cache in Cloud Storage so a restart does not pay for the same compile twice. The interface underneath is the Interim Framework Runtime from the OpenXLA project.
That is a scheduling decision with a real trade. One controller is far easier to program than a set of hosts negotiating with each other. It is also a single point of coordination, and Google answers the obvious objection with the elastic path rather than with an availability claim. The setup utility itself is public, so a platform team can read the pathways-utils code rather than take the marketing deck’s word for it.
A failed slice shrinks the job instead of killing it
When hardware drops, Pathways resizes the training job down to N-1 slices using host-memory snapshots and keeps the client alive. The job continues at lower throughput instead of restarting from the last checkpoint.
That changes what a fault costs. On a fixed-shape job the arithmetic is brutal, because the whole fleet waits on a restart. On an elastic job the cost is a throughput step, and the team’s actual job becomes deciding how far down the step can fall before a restart is the cheaper answer. Colocated Python sidecars put data loading and checkpointing on the TPU hosts in shared memory.
The customer figures are the tell. A hundred million chip-hours a quarter from one account is not a pilot. At that volume the queue and submission layer is a genuine operational cost, and Google is selling the layer it built for its own Gemini work. ZenteiQ, a scientific foundation model shop, runs elastic pre-training with MaxText and single-controller reinforcement learning on Trillium slices.
This is a TPU answer to a problem every fleet has
Pathways is Cloud TPU only. Every claim above is scoped to v5e, v5p, v6e and TPU7x. A GPU fleet gets nothing from the release, and Google’s GPU scheduling story runs through a different stack. If your accelerators carry a green badge, this is a data point about how hyperscalers manage large fleets, not a deployment option.
The second limit is Slurm. A preview integration with Cluster Director extends Pathways to Slurm TPU clusters, with unified multi-slice JAX, elastic training and interactive notebooks. Preview, not general availability. Most pre-training outside the hyperscalers still submits through Slurm, so a lab that wants this runtime and lives in Slurm is still waiting, and the wait is the part of the announcement Google understates.
The general lesson holds either way. The runtime that decides how accelerators share one job is now a measurable slice of the bill, which is why the economics of rented capacity keep moving away from the chip. Google already showed what a Kubernetes control plane looks like when it runs the accelerator fleet itself, and the NVIDIA control plane release made the same point from the other direction. The scheduler is the product.
Three questions for your own accelerator fleet. Does your runtime survive a failed slice without restarting the world. Where does the compile cache live when the controller process dies, and what does a cold start actually cost you. And what throughput step are you willing to accept before a restart becomes the cheaper option.
Documentation for the runtime is at Pathways on Cloud, and the JAX release line is tracked at jax-ml/jax. The announcement itself is on the Google Cloud AI infrastructure blog.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.