Disaggregated Inference Added a Latency Tax. More Bandwidth Was Not the Fix.

Prefill and decode are different jobs. Prefill reads the whole prompt at once and is bound by compute. Decode emits one token at a time and is bound by memory bandwidth. Running both on the same GPU means one of them is always waiting on the other’s bottleneck. Prefill-decode disaggregation puts them on separate engine instances so each can be sized and scaled on its own terms, and it has become common practice for serving long prompts at high concurrency.

The price of that split is a KV cache that has to move. When prefill finishes, the blocks it computed must reach the decode instance before generation begins. vLLM moves them through the NVIDIA Inference Xfer Library, or NIXL, which transfers KV blocks directly between device memory over a remote direct memory access fabric. On AWS that fabric is Elastic Fabric Adapter, and the transfer uses GPUDirect RDMA so the data never stages through host memory.

The setup work was sitting in front of the first token

In the pull-based connector the decode instance does the work. It receives the block identifiers from the router, allocates destination blocks, prepares the transfer and issues the read. Those steps run in sequence after prefill and before decoding, and they run inline on the decode worker’s engine loop. Token generation waits on all of it.

The push-based connector moves that work off the loop. Descriptors are prepared on a background thread while prefill is still running, and the transfer is submitted without blocking the engine, so the decode worker is never starved. The steps are the same. Only their position on the path to the first token changed.

On a Qwen3-32B deployment with one prefill instance and one decode instance across Amazon EC2 P5en instances, the published numbers show time to first token falling from about 1,744 milliseconds to about 540 at 8 requests per second with 16k input tokens. Time per output token fell from roughly 10.4 milliseconds to about 7. At low load and short prompts the two modes are close, and the gap opens as concurrency and prompt length rise, which is the behaviour you want from a fix aimed at the critical path.

Disaggregation is a trade, and the connector makes it visible

Push mode requires the decode instance to be chosen before prefill completes, because the prefill side needs somewhere to push to. Pull mode can defer that choice until after prefill, which lets a router pick the least loaded decode node. The vLLM team keeps both connectors for that reason and points workloads that need bidirectional KV transfer or load-aware placement at the pull path. If your router already makes placement decisions from live metrics, this feature asks you to give something up.

The feature was built and validated on AWS Trainium clusters and then contributed to vLLM as accelerator-agnostic, which is why it lands in the general release rather than in a vendor build. That detail matters if you run disaggregated serving across a mix of accelerators. The setup guide is in the disaggregated prefill documentation, and the change is a connector swap plus the routing metadata the proxy needs.

Related reading. The reason the KV cache is the thing you engineer around is that HBM cannot hold a long context. On the Rubin generation vLLM has already shown substantially higher throughput per GPU, and the GPU rental market sets what an idle decode node costs you.