KV Cache Offload Works. Your East-West Network Sets the Ceiling.

The most useful number in a KV cache deployment is not the speedup. It is how much bandwidth each GPU node can pull from the storage tier holding its cached context. A joint design from Supermicro, IBM and NVIDIA Dynamo makes that plain, and mostly by saying it out loud in the documentation.

The components are ordinary parts assembled in a specific order. Supermicro Petascale all-flash storage servers on AMD EPYC 9005 processors hold the cache. IBM Storage Scale serves it as a shared parallel file system. NVIDIA Dynamo routes requests to the node holding the matching context, splits prefill from decode and moves cached blocks with its own transfer library. Spectrum-X Ethernet carries the traffic. Dynamo places context across a tier model that starts in HBM, drops to CPU DRAM, then local SSD, then pod flash, and finally the external shared tier.

The numbers come from June, and they measure a ceiling

The IBM Redbooks reference architecture is dated 5 June 2026 and was last updated on 20 August. The November release packages it for order. That matters for how to read the 56x, because the comparison is a single request at 130,000 tokens against a GPU server with no external cache store at all. The launch release adds the more interesting framing. Time to first token grew by 0.377 seconds going from 30,000 to 130,000 tokens with the cache, against 29.55 seconds without it.

That second pair is the one to write down. Nearly flat latency as prompts grow is the property that makes long context affordable. A ratio against no cache at all is the correct control and a poor forecast, because your production stack already keeps hot context in HBM and DRAM and only sends the cold tail to the shelves. The 56x is the ceiling of what the shared tier can add, not what a working deployment will notice.

Contention costs throughput, and the fabric decides how much

The concurrency benchmark is where the numbers get useful. Across 200 requests the cached configuration moved from 0.19 to 4.26 requests per second, a 22x improvement, and cut total processing time by 95 percent. Then the authors added four client nodes pushing 200 GB/s of competing network traffic to simulate a busy multi-tenant fabric. Throughput held at 3.6 requests per second and finished the same 200 requests in 55.56 seconds, which is 18x over recomputation and a real drop from the clean run.

Losing roughly 15 percent of throughput to a noisy neighbor is survivable. It is survivable because this rig had room. The Redbook is explicit that the storage tier carried unused headroom and that additional 400GbE client-side links would lift retrieval performance further. Read that sentence twice. The bottleneck in a KV cache design sits between the GPU node and the switch, not inside the array.

Ask for the fabric ratio before the multiplier

The question that decides a purchase is what bandwidth reaches one GPU node, and what that number becomes when you add nodes. A design that gives a node a quarter of what the array can deliver will behave like the constrained test, not like the clean one. Two details belong on every vendor questionnaire. Was HBM and DRAM caching active during the measurement, or was the tier isolated to show its own contribution. And at what concurrency and what prompt distribution does the reported ratio hold.

This is the same seam we covered when SambaNova and Supermicro put prefill and decode on separate silicon, and when vLLM shipped a push based KV transfer between the two halves of a request. KV cache offload turns compute into bytes on a network, and so does every one of those designs. Compute gets cheaper every generation. The fabric bill arrives once.

Three questions for the next design review. What is the client-side bandwidth per GPU node, in bytes per second, on the fabric you actually own. Which cache tiers were enabled when the vendor measured its ratio, and does the answer match your production topology. And if a neighboring tenant saturates the switch, what is the throughput you can still promise your users in writing.