llm-d Hit 1.0 and the First Feature Is a Fair Queue

The v1.0.0 notes open with traffic control. A probabilistic admitter, program aware fairness, turn priority scoring, and controlled load shedding under overload. Release notes get written in the order of pain, and this one starts with the queue rather than the kernel.

llm-d runs in front of vLLM or SGLang on Kubernetes and turns a set of replicas into one serving pool. Each request goes to the replica that already holds its prefix, context spills into a cache tier below GPU memory, and prefill can run on different machines from decode. The llm-d 1.0 announcement describes the same request path running across a mixed fleet of TPUs and GPUs behind a single endpoint.

A round-robin balancer breaks the moment agents arrive

A year ago most inference traffic was chat. One prompt in, one response out, and a balancer that spread requests evenly was close to correct. Agent workloads behave differently. They share long prefixes, fan out across tools, and arrive in bursts that drag a whole replica set into the same expensive prefill.

The fix is admission, not more replicas. Flow control in 1.0 scores each request, protects the latency sensitive ones, and sheds what the fleet can’t serve rather than letting every caller wait. The operator consequence is that admission policy becomes an object you own. A rate limit on the API gateway no longer describes what your cluster will accept.

The upgrade path is where 1.0 sends the bill

Two changes cost time. The router configuration API graduates to v1 on the release, so configs written against the 0.x schema need rewriting rather than a version bump. The project is also deprecating its own inference images, keeping those streams only long enough for hardware partners to move onto upstream builds from vLLM and SGLang. The ROCm image went in this release, replaced by a pinned vLLM build.

What you get for the work is operations. DisaggregatedSet treats prefill and decode replicas as one logical unit, with coordinated rollouts and readiness gated on a pod snapshot. Router high availability with a hot standby rounds it out. The notes also say plainly that not everything in the tracked goal was finished, which is a useful thing for a 1.0 to admit.

Hit rate is the number worth arguing about

The economics sit in cache reuse. Snap reports prefix cache hit rates of up to 75 to 80 percent after putting the router behind an Envoy based service mesh, and it says so in the announcement. The project benchmark page claims 3x higher output throughput and 2x faster first token.

Treat those as the vendor numbers, because they are. Then look at yours. A pool that recomputes half its prefixes pays for tokens that never appear on a latency chart. Routing decisions, not GPU choice, set that number.

Three questions for your next architecture review. What is your prefix cache hit rate today, measured at the router rather than guessed from memory? Which of your tenants shares a fleet with a batch job that arrives in bursts? And if the router restarted mid rollout, how much context would your agents rebuild?

Related reading. Our earlier pieces on routing as the scheduler problem and on the latency tax inside disaggregated inference cover the parts this release assumes you already run. The llm-d repository and the CNCF project page carry the current state.