Microsoft’s Maia 300 Is an Inference Chip, and Tokens per Megawatt Is Why It Matters

Microsoft’s Azure Maia 300 is a rack, not a card. The unit the company is selling is 96 accelerators inside a 167 kW envelope, and the whole design follows from that number. Microsoft published the figures on October 12 and described the chip as the third generation of a program it started in 2023.

The chip is inference-first, and the rack is where the design happens

Maia 300 packs 96 accelerators across 24 trays of four inside a single 167 kW rack. Microsoft folded the host processor, memory, storage and cooling into each tray, so there are no separate host trays and the power and thermal budget is managed as one system. The chiplet layout separates compute, memory controllers and input-output, and the compute chiplets are built on TSMC’s N2P process. Each accelerator carries eight stacks of HBM4 memory, 18.0 TB/s of peak bandwidth and a 950 watt thermal design power.

Two changes matter for the software above it. Memory moves from a uniform layout on the chip to a locality-aware non-uniform layout across chiplets, so the stack can keep data near the compute that uses it. Microsoft also added narrower numeric formats with hardware conversion, including an enhanced MXFP4, which cuts the bytes moved for every token. Liquid cooling removes more than 80 percent of the rack thermal load from the accelerators, the voltage regulators and the switch silicon.

Microsoft says a full rack delivers up to 7x the rack-scale token throughput of Maia 200 and up to 2.5x the performance per watt. Against leading GPU-based systems, the claim is 40 percent more tokens per megawatt. The scale-out path runs over standard Ethernet to a superpod of 12,288 accelerators.

Power is the constraint, so power is the scoreboard

The interesting part is not the silicon. It is the denominator. Microsoft is not competing on peak floating-point throughput, where Nvidia leads and where training still lives. It is competing on how much useful output one megawatt of secured power can produce.

That is the right contest for the market Microsoft is actually in. Microsoft runs Copilot, GitHub Copilot, Microsoft Foundry and its own models, and nearly all of that is inference. Inference is a volume business against a fixed power bill, so tokens per watt sits closer to gross margin than any benchmark score. Maia 200 is already deployed across the United States, Europe and Australia, and Microsoft says it delivers up to 1.7x more tokens per watt than leading GPU infrastructure for features in Teams, Copilot and Cowork.

Two caveats keep this honest. Maia is not rented to outside customers, unlike the comparable chips from Google and Amazon, so it is a cost play rather than a new revenue line. And Microsoft would not give a timeline for the multi-gigawatt deployment, which its own executive vice president Scott Guthrie described only as starting in 2027.

The analyst read is that the incumbent does not lose by default. “I don’t expect Maia to slow Nvidia or AMD deployments across Azure,” wrote Matt Kimball of Moor Insights and Strategy. “AI demand is growing too fast for any single supply chain.” Evercore ISI estimates that Maia will account for about 10 percent of Microsoft’s AI chip spending by early 2027, according to CNBC.

What to ask any vendor who now sells compute by the megawatt

If your own capacity planning runs in megawatts, Microsoft just handed you a comparison you can apply to every accelerator vendor. Ask what a fully loaded rack draws, what it delivers per megawatt, and how much of that number is a vendor claim rather than a measured workload on your own traffic.

Three questions are worth putting to any vendor this quarter. What is your tokens per megawatt on my workload rather than on a benchmark tuned for a launch. What is the power envelope of a fully populated rack once cooling is included. And what is my exit if your serving stack does not fit the scheduler I already run.

Related reading. For the serving layer this feeds, see how vLLM reached 7.8x more per-GPU throughput on Nvidia’s Rubin, and why the cheapest inference on the market is a classifier, not a chat model.