Fireworks Says the Token Price Is a Decoy. The Cache Hit Rate Sets the Bill.

The most expensive part of an AI workload is not the tokens you buy. It is the tokens you buy twice.

Rob Ferguson, a vice president at managed inference provider Fireworks AI, made that case to The Register in an interview published Saturday. Token price is not the cost of the work, he argued. Cost per task is, and almost nobody can see it.

Ten thousand idle GPUs bill exactly like ten thousand busy ones

Ten thousand GPUs, he said, represent a billion dollars of capital, or about $100,000 a card. Spread across a four year life that is roughly $2.85 an hour per GPU before it draws a watt.

That is the real case against owning hardware, and it is about utilization rather than price. Fireworks routes requests through a system it calls Nexus, sending each to whichever model it judges suited, and it competes with Together AI, Baseten and DeepInfra on the same promise. An agent fleet that peaks for three hours and idles after cannot fill a cluster it owns.

Where load is steady and the model is pinned, the argument weakens. A service running one set of weights at 85 percent utilization for three years is not paying for idle silicon. It is paying for capacity it uses.

Most of the claimed saving lands in the cache, not the router

Here is the part of the interview that survives contact with an invoice. Ferguson said a Fireworks customer, a data science company, was caching about 90 percent of its tasks. In a comparison the company ran against Anthropic’s Opus, he said, the model hit a cache 30 percent of the time and was doing AI rework constantly.

Fireworks has not published that comparison, so read the 30 percent as a claim about a competitor. The arithmetic underneath is public. On Anthropic’s prompt caching pricing a cache read bills at 10 percent of the base input price, so the effective cost of an input token tracks the hit rate almost linearly.

At 90 percent hits the effective cost is about a fifth of the uncached figure. At 30 percent it is about three quarters. The gap is a 74 percent cut, most of the six to eight times Ferguson claims, and it happens before a request is routed anywhere.

Hit rate belongs to the workload, not the vendor. Agent loops with a fixed system prompt and a long history cache well. One-shot requests with a fresh prompt every time cache at zero. Routing fights that a little, because a cached prefix belongs to the model that wrote it, so good routers send similar task classes to the same model.

A model portfolio is a second platform, and the invoice omits it

The routing effect shows up in Fireworks’ own case study with Factory. A Kimi K2.7 model ran a task at roughly 20 percent of frontier cost and a MiniMax model at roughly 6 percent, and Factory’s router cut cost per average task by 30 to 40 percent.

The cost the invoice never shows is the platform you now own. A portfolio means an evaluation harness that scores every model on your own tasks, prompts that survive a swap, version tracking, and a caching strategy re-earned on every model change.

There is an incentive worth naming too. A routing vendor earns more when more traffic passes through it, so the advice to use every model also grows the account. That does not make it wrong. It makes it worth pricing against your own numbers.

The useful move is smaller than the headline. Instrument cost per task instead of cost per token, split into first attempt, cache hits and rework, then ask how much of your traffic really needs a frontier model. We looked at why GPU rental prices rise while token prices fall, and at what KV cache offload costs in east-west bandwidth.

Three questions to run on your own numbers. What share of your input tokens hit a cache last month, and who owns that figure? If every task went to the cheapest model that clears your bar, what would the bill be? And if the saving is real, which team absorbs the evaluation work that makes routing safe?