IBM, NVIDIA and Supermicro packaged a shared KV cache tier this week, and their own documentation names the limit. It is the network, not the storage.
- Cached context sits on an all-flash tier running IBM Storage Scale, moved by NVIDIA Dynamo over Spectrum-X Ethernet.
- The 56x time to first token figure comes from an IBM Redbooks architecture published in June, not a November benchmark.
- Across 200 requests the cache took throughput from 0.19 to 4.26 requests per second, a 22x improvement.
- With four clients pushing 200 GB/s of competing traffic it held 3.6 requests per second, still 18x faster than recomputing.
- The Redbook states that the storage tier has headroom it cannot deliver, and that more 400GbE client-side links would unlock it.
- Ask for bandwidth per GPU node, not the multiplier, when you size the design.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.