Executive Summary
AWS shipped its GPU-aware inference router as a single managed EKS add-on, and the more consequential detail is what sits underneath it. Amazon SageMaker HyperPod Inference Gateway is built on the open Kubernetes Gateway API Inference Extension and names llm-d as its scheduler. For two years platform vendors shipped bespoke inference routers. AWS shipped conformance to a specification instead, and that turns the routing layer above your model servers into a standard.
The measured gains are real but conditional. Against a Kubernetes round-robin baseline on the same replicas, AWS recorded first-token latency at P99 down 97 percent on mixed GPU generations and 98 percent under bursty traffic, with one model gaining 50 percent more throughput. On a uniform fleet under steady load the gateway and round-robin performed the same, and AWS says so plainly. The verdict is narrow and useful. Intelligent routing pays where production fleets are uneven, which is almost everywhere.
Round-robin is a fine way to spread stateless web traffic across identical replicas. It is a bad way to route large language model inference, because the replicas are not interchangeable. One pod may be holding a warm prefix cache. Another may be part-way through a long generation with its KV memory nearly full. A third may already have the LoRA adapter your request needs sitting in GPU memory.
Default Kubernetes load balancing sees none of that. Requests pile up behind the pods that are already busy while idle capacity sits unused, and first-token latency, the number users actually feel, moves first. AWS puts the spike at four seconds or more during a burst.
AWS put the fix behind one add-on install
The Amazon SageMaker HyperPod Inference Gateway installs as a single EKS managed add-on called amazon-sagemaker-hyperpod-inference, on the HyperPod infrastructure a team already runs. It is made of three parts. Envoy Gateway terminates TLS and exposes one private endpoint per cluster. A Body-Based Router reads the model field out of an OpenAI-compatible request body and chooses the model pool. An Endpoint Picker then scores live model pods against Prometheus metrics and picks one.
The picker weighs five signals, each with a configurable weight. KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and running requests. The whole topology is declared in one custom resource, InferenceGatewayConfig. There are no sidecars, no service mesh, and no application changes. Any OpenAI-compatible server works, including vLLM, SGLang, and TGI.
The routing layer standardized, and AWS joined it
Two details carry more weight than the feature list. The gateway is built on Gateway API Inference Extension, an official Kubernetes project that extends any Gateway API proxy into an inference-aware router through Envoy’s external processing protocol. And the sample configuration sets the scheduler to llm-d, the distributed inference stack that KServe and Red Hat have been building on.
That is a hyperscaler adopting the community endpoint-picker pattern rather than inventing a private one. Endpoint pickers, prefix-cache-aware routing, and declared model pools are now a specification an operator can carry between clouds. The days of a different routing API on every platform are ending, and the platforms that already speak this one have a head start.
The numbers hold only when the fleet is uneven
AWS benchmarked four models from 8B to 235B parameters on p5.48xlarge and g5 instances, with traffic going through internal load balancers on the same path a production request takes. First-token latency at P95 and P99 fell 97 and 97 percent on a mixed-generation fleet running Llama-3.1-8B, and 98 and 97 percent on Qwen3-32B, which also gained 50 percent more throughput. Bursty traffic on Llama-3.1-70B fell 94 and 98 percent with 12 percent more throughput. Shared prompt prefixes on Llama-3.1-8B fell 26 and 43 percent.
Then the honest row. On a uniform fleet under steady traffic running Qwen3-235B, the gateway and round-robin were comparable, and AWS labels the gap as within run-to-run variance. If your GPU estate is one instance type, one model, and even demand, this changes nothing for you. If it is mixed generations, multi-model, or bursty, and it still sits behind a plain Service, you are paying for that in latency.
Tier 2, a Global Inference Router that adds cross-cluster failover, global rate limits, and cost-aware traffic shaping, is not out yet. When it lands, routing decisions move from the cluster to the fleet. Watch whether the other clouds adopt the same specification or ship their own.
Related reading. Our look at why Kubernetes for AI is not Kubernetes for apps covers the scheduling assumptions that break under inference. And the v1.37 scheduler work explains how the control plane is being rebuilt around accelerators.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data centre economics. No vendor spin.