Executive Summary
SambaNova and Supermicro will build and sell inference systems that put NVIDIA GPUs and SambaNova SN50 reconfigurable dataflow units in the same rack, with an initial plan for 10,000 SN50 chips. The design routes the two phases of a request to different silicon, prefill to the GPUs and decode to the RDUs. Supermicro handles integration, validation, delivery and on-site support.
The commercial argument is not about speed. It is about where the capacity goes. SambaNova says the RDUs deliver high throughput on large models without extreme rack power density, so operators can add capacity inside buildings they already run instead of waiting on construction constrained by power and cooling. That claim is plausible and entirely unmeasured in the announcement. No throughput, latency, power or price figure was published for the combined systems. The number that decides the deal is watts per token, and nobody printed it.
Every operator who needs more inference capacity in 2027 has the same problem. The accelerators are orderable. The building is not.
Power and cooling set the ceiling. Interconnection queues set the schedule. SambaNova and Supermicro are pitching a way around the building.
Inference splits into two phases with opposite appetites. Prefill reads the prompt and runs large matrix multiplies across the whole context, which is compute bound and is what a general purpose accelerator does well. Decode emits one token per pass, and every pass has to stream the weights, so decode is bound by memory bandwidth and leaves most of the GPU’s arithmetic idle. Agentic workloads spend the bulk of their wall clock there, because an agent makes many calls and each call produces a long answer.
So the split is not new. vLLM ships disaggregated prefill, and disaggregated inference has routing and cache layers built on the same seam across the serving stacks. We covered the latency tax that arrives when the KV cache has to cross a network between the two halves. What is new is doing it in hardware you buy, on the same purchase order as the GPUs, with a server vendor carrying integration and on-site service.
The retrofit case is real and it rests on one unpublished number
SambaNova says the RDUs serve large models at high throughput without needing extreme rack power density, so a colo operator can drop them into an existing hall. You add inference capacity without waiting for a build-out, and you extend the life of GPUs you already own because they keep doing the compute-bound half.
That is the right argument to make in 2026. New capacity is not limited by wafer supply at this point. It is limited by megawatts, and by how long a substation takes to approve.
What the announcement does not give you is the number that tests it. Watts per token for the combined system, throughput at a stated model size and batch, and the draw of a loaded SN50 rack. Without those, the retrofit case is a floor plan rather than a business case. Every colo contract is written in kilowatts per rack, and a heterogeneous rack moves that figure in both directions at once.
A heterogeneous rack is two of everything to operate
Two accelerators in one rack means two driver stacks, two firmware cadences, two failure domains and two supply chains. It also means the KV cache crosses a link that did not exist in a single-vendor cluster. Our earlier look at vLLM’s KV transfer work found that the first version of disaggregation added latency before it removed any.
The operational question is whether a team that already struggles to keep one GPU stack patched wants a second. SambaNova says its engineers will tune models alongside Supermicro’s deployment crews, which is a services answer to a software problem. It works for the first ten customers. It does not obviously work for the hundredth.
Sovereign clouds go first because waiting is their worst option
The rollout order says a lot. Neoclouds and sovereign clouds get the first systems, enterprises come later. That is backwards from where the volume is, and it is correct for a product that has been announced but not benchmarked.
A neocloud is capacity starved and can absorb a new architecture because it owns its halls and its engineers. A sovereign program has a mandate that outranks a comparison spreadsheet. Both can take a bet on unproven silicon. An enterprise buying for a five year depreciation cycle cannot, and knows it.
The shape of this deal is right. Splitting prefill from decode is the correct response to a workload that is now mostly decode. The open question is whether the efficiency claim survives a power meter, and that is answerable with one published table.
Three questions for your own capacity plan. What is your current watts per token at production batch size? How much of your 2027 inference demand could run in halls you already lease? And if a second accelerator family arrived tomorrow, who on your team would own its driver lifecycle?
Related reading. Our analysis of llm-d 1.0 and its fair queue covers the software layer making the same split, and our piece on Cerebras and Gimlet covers why two kinds of silicon do not beat one on operational simplicity.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.