Executive Summary
DeepInfer is the inference engine running DeepSeek’s own production services, published on 20 October under Apache 2.0. It supports one model, DeepSeek-V4.1-Flash, on two chip families, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It ships tuned kernels, MoE expert parallelism with its own load balancer, draft-token decoding verified against a confidence signal, and a batch scheduler that overlaps CPU work with device execution. The repository took roughly 17,600 stars and 1,800 forks in its first two days.
The README also lists everything a complete inference service needs and this release leaves out, and that list is the more interesting document. A KV cache system. Prefill and decode disaggregation. Load balancing across data-parallel replicas. Every one of those decides what a token costs once you run more than one replica. DeepSeek published the engine and held back the fleet manager, which is a fair trade for a company that owns both ends of its own stack and a real gap for anyone else.
DeepSeek published the part of its inference stack that anybody can now copy. It kept the part that is hard. Both are in the same README, a few screens apart.DeepInfer is Apache 2.0 and you build it from source. GCC 13 or newer, Python 3.12, and then either CUDA 13 or Huawei’s CANN 9.2 depending on the card you own. The repository had four commits and no release tags when I read it, which tells you how fast this went out the door.
One model and two chip families is the entire support matrix
The supported model list has exactly one entry. The hardware list has two. That is a narrowing decision, not an oversight, and it is where the performance comes from. A general serving engine spends its code on every architecture’s quirks. A single-model fork can spend all of it on one set of tensor shapes.The benchmark numbers in the repo show what the shapes look like. Prefill was measured on 4 SM100 GPUs and decode on 64, each inside a single scale-up domain. That asymmetry is the workload. Prefill is a dense matrix multiply and spreads across a few cards. Decode is latency bound and eats the whole fleet, which is why the decode side is where the cost per token actually lives.Ascend support is not cosmetic. There are separate requirements files for the NVIDIA and Ascend paths, and the build switches with one environment variable, which means the Huawei path is maintained next to the Nvidia one rather than bolted on afterwards.
The README names the three pieces it did not ship
The README has a section titled required for a complete inference service, followed by the words not included in this release. Three items sit under it. A KV cache system. Prefill and decode disaggregation, with a note that most of the engine work for it is already implemented. Load balancing across data-parallel replicas for both prefill and decode.Read those three against the way inference bills work and the shape of the trade becomes obvious. The cache tier decides how many replicas you can retire. Disaggregation decides whether prefill and decode share a card or get their own. Replica load balancing decides whether your 64 decode GPUs are all busy or only the hot ones are. Those are the knobs that move cost, and none of them are in the tarball.There is a smaller portability caveat too. Batch-invariant inference and deterministic sampling, the two features that make a model return identical output for identical input, are flagged NVIDIA only in the README. The determinism story does not travel to Ascend yet.
A vLLM fork, specialized until generality stopped being the point
The acknowledgments settle what this is. DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. The kernel work sits in separate DeepSeek repositories, including FlashMLA for attention and DeepEP for expert communication, and the default checkpoint is DeepSeek-V4.1-Flash-1017 on Hugging Face.That is not a criticism. It is the honest description of how the fastest inference in the world gets built at the moment. vLLM and SGLang give you one engine for a hundred models. DeepSeek wanted one model on two accelerators at a cost nobody else could match, so it took the open engine, cut it to the model, and kept the tuning in house until this week.The question for your own platform is how many entries your model zoo has. If the answer is one, a fork like this is the cheapest throughput you can buy and you will happily write the cache tier yourself. If the answer is twelve, you have just been handed a second serving stack to operate, and the throughput gain has to clear that cost.Ask three things before you clone it. Which of the three missing pieces does your platform already own, and is it any good? Does your batch composition need the deterministic mode, and if so does that pin you to Nvidia? And when the next DeepSeek checkpoint lands with different shapes, who re-tunes the kernels?
Related reading. We covered why llm-d put a fair queue in front of the GPUs at 1.0, and what happened when vLLM let coding agents write its kernels.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.