HBM Cannot Hold the Context Window, and the Memory Industry Just Said So

Every token an inference server generates leaves behind a key and a value vector, and the model attends to those vectors to produce the next token. That store is the KV cache, and it grows with context length and with the number of users on the box. It has lived in HBM because nothing slower was fast enough.

HBM is the smallest and most expensive memory in the machine. When the cache fills it, a server has three bad options. Evict and recompute, which burns the accelerator it was trying to save. Refuse the request. Or spill the cache to a slower tier and accept the latency. This week, three products attacked the third option at once.

CXL turns a tray of DRAM into shared capacity

Marvell announced the Structera X 3406 and 3506 on October 12. Each part carries six DDR4 or DDR5 channels, up from four, and a PCIe 6.0 x16 host link of up to 256 GB/s. The 3406 drives 18 DDR4 RDIMM slots for up to 9TB. The 3506 drives 12 DDR5 slots for up to 6TB.

The number that matters is the host count. One controller can present its DIMMs to up to four servers at once, so a tray of memory becomes a shared pool rather than a private expansion card. Inline LZ4 compression runs at up to 250 GB/s and Marvell claims it unlocks two to three times more effective capacity. Against its own previous generation, the company reports time to first token improved by 40 to 50 percent at the same context length. CXL is the interconnect that carries the traffic, and Meta and the memory makers Samsung and SK hynix supplied supporting statements. The parts are expected to sample in the first quarter of 2027.

The software layer is moving just as fast

Tensormesh, the team behind the LMCache project, released its Platform on October 14. It tiers the KV cache across accelerator memory, host CPU memory and storage, deploys on Kubernetes, and adds multi tenancy and a form of caching that reuses computed blocks anywhere in a prompt rather than only at the start.

Inworld launched an inference service the same day with a router at the center of it. The router hashes each block of a prompt, tracks which replica holds which blocks, and scores every replica on reuse and load before it sends a turn. Inworld reports cache hits rising from 25 to 47 percent on a replay of production traffic, and cached input priced up to 99 percent below fresh input. The mechanism is the same one vLLM has pushed with automatic prefix caching. The change is that the cache is now a system-level resource with its own hardware, its own router and its own line on the bill.

What to ask before you buy the tier

Two caveats keep this honest. CXL memory is not local DRAM, so a cache read from it carries a latency the HBM path does not. And every performance number here is a vendor claim. Marvell’s gain is measured against its own previous generation, not against a rival and not against your workload, and the parts sample in 2027, so nothing is available to test yet.

Three questions for an inference budget. Where does the KV cache leave HBM in your stack today, and what does that cost you in recompute. What is your cache hit rate under real traffic rather than on a warm run. And if you add a router that pins conversations to specific replicas, what does that do to the autoscaling assumptions in your scheduler.

Related reading. For where this compute is rented and how it is priced, see the GPU cloud market at 323 providers and NVIDIA making Run.ai open source as the GPU scheduling layer above it.