Executive Summary
Mistral released the weights for Mistral Large 4 on 29 October, three weeks after a public preview. The model carries 1 trillion parameters with 49 billion active per token, a 1M token context window, and ships in FP8 and NVFP4 checkpoints. vLLM shipped day-0 support the same day, with decode context parallelism lifting KV cache capacity by about 7.9 times on eight GPUs, a built-in speculative drafter that accepts about 2.9 tokens per step, and prefill context parallelism that cuts time to first token by up to 1.6 times.
Open weights rarely mean runnable, and this release makes the arithmetic visible. The checkpoint format decides the GPU count before any benchmark does, and the context window decides how many concurrent requests one replica can hold. Mistral’s own comparison is a token count, 47 percent fewer tokens per task than GLM-5.3. Operators who size on headline throughput will oversubscribe the cache and find out under load.
The weights dropped on 29 October. By the end of that day the interesting number was not the one trillion parameters. It was 7.9, the multiplier vLLM reported for KV cache capacity once decode context parallelism is spread across eight GPUs.
Mistral Large 4 is a mixture-of-experts model with 72 layers, 128 routed experts and a top-4 routing pattern, plus one shared expert. It holds about 1 trillion parameters in total and 49 billion active per token, 52 billion once embeddings and output layers are counted. Attention is multi-head latent attention with a rank of 512, and a sparse indexer selects the top 2,048 keys per query instead of attending across the whole 1M token window. The weights are on Hugging Face in two official formats.
The checkpoint decides the GPU count before the benchmark does
Mistral opened the weights in two formats, FP8 with block-scaled weights, and NVFP4 with NVFP4 experts and FP8 attention. In vLLM’s own recipes, the FP8 weights run at tensor parallel 8 and the NVFP4 weights run at tensor parallel 4 on Blackwell. Eight GPUs against four is not a tuning preference. It is the difference between two pools of capacity and one, decided by the format you downloaded.
Both checkpoints use an FP8 latent KV cache by default. That matters more than the weight format for concurrency planning, because the cache is what a long conversation consumes.
Sparse attention moved the bottleneck into the cache
Every decode step keeps a running cache of the keys and values the model has already seen. That cache grows with the prompt and the conversation, not with the weights. At a 1M token window the cache per request is large enough that concurrency becomes a memory question before it becomes a compute question.
The 7.9 times figure is decode context parallelism spreading that cache across eight GPUs instead of one, so a single request can hold far more live context. Read it correctly. The capacity is bought with GPUs, not created by the algorithm. What the algorithm did was make each token cheaper to cache, through cross-layer KV sharing, an FP8 latent cache, and attending to 2,048 keys instead of all of them.
Host-memory offload is the rest of that argument. HiSparse moves sparse KV to host memory and now supports cross-layer sharing, and NIXL handles prefill and decode on separate nodes. Both keep a 1M window alive on hardware that cannot hold it in HBM alone, and both show up as time to first token.
Day zero is now a procurement question
Support for a new architecture the day the weights land used to be a boast. It is the baseline across the serving stacks now, so the differentiator is which architecture features an engine has actually implemented. This release asks for a lot of them, including the speculative drafter built into the model, the sparse indexer, the vision encoder for multi-image prompts, and expert parallelism with MXFP4 experts. vLLM also reports day-0 support on AMD’s Instinct MI350 series under ROCm.
Two figures make the economics concrete. Mistral puts Mistral Large 4 at 47 percent fewer tokens per task than GLM-5.3, a better cost measure than tokens per second. The model card lists $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens on Mistral’s own API. Self-hosting replaces all three with a GPU lease, a power bill and a person.
The practical consequence is a version pin. The engine that can serve the architecture you bought is the engine you will run, and upgrading it becomes a cluster change rather than a library bump. Teams that treat the serving stack as fixed infrastructure wait on someone else’s release to use a model they already paid for.
Three questions for your inference fleet. What is your KV cache capacity at the context length your product actually advertises? Does your capacity plan count GPU memory for weights only? And when the next open model lands, which engine on your platform gets it first, and how many days does that take?
Related reading. The pool held 681,280 tokens and the window said 1,048,576, and llm-d hit 1.0 and the first feature was a fair queue.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.
[…] a runaway job and the person who can kill it. The same arithmetic shows up one layer down, where the KV cache decides who can afford to serve a large model at […]
[…] the tokens a second it was sized for and the answer is another node rather than a tuning pass. The KV cache arithmetic we ran for a trillion parameter model is the same arithmetic at a lower read rate. The host CPU […]