The Inference Cost Paradox: Token Prices Fell 10x. Your Bill Tripled.

Per-token prices are collapsing faster than almost any technology input in history, and enterprise AI bills keep going up anyway. That is not a contradiction. It is three structural forces doing exactly what they are built to do, and every one of them is under your control, not your vendor’s.

By Abhimanyu Singhal · July 10, 2026 · 12 min read

In my last post on effective context windows, I argued that the most quoted number in enterprise AI, the context window, tells you almost nothing about what a model can actually do. This post is about the second most quoted number: the price per million tokens. It tells you almost nothing about what you will actually pay.

Here is the paradox in two sentences. The price of LLM inference at a fixed capability level has been falling roughly 10x per year. a16z christened the trend “LLMflation,” and Epoch AI’s more careful analysis found declines ranging from 9x to 900x per year depending on the benchmark, with a median around 200x per year for data since January 2024. Over the same period, enterprise spend on LLM APIs went from $3.5 billion to $8.4 billion in six months (Menlo Ventures’ mid-2025 figures), and total enterprise AI investment tripled in a year to $37 billion.

If a normal input got 10x cheaper annually, you would expect bills to shrink. They are not shrinking. In most of the teams I talk to, they are growing faster than the workloads that justify them. The reason is not vendor greed, and it is not waste in the trivial sense. It is that per-token price is the wrong unit of account. You are not buying tokens. You are buying task completions. And cost-per-task is governed by three forces that grow consumption faster than prices fall, all of them architectural, all of them yours to manage.

The wrong number on the invoice

The parallel with the context-window post is exact. There, the spec sheet printed a capacity (1M tokens) that told you nothing about capability (effective context). Here, the pricing page prints a rate ($X per million tokens) that tells you nothing about your cost driver (tokens consumed per completed task). One is a price. The other is price times volume, and volume is where the game is being played.

Three forces inflate that volume. Take them in order of how badly they are underestimated.

Force 1: Nobody banks the savings

LLMflation is real. If your workload was well served by a 2024-frontier model, you can run that same capability today for pennies on the dollar. Almost nobody does.

Menlo’s data shows why. When a new frontier model ships, builders migrate to it en masse. Within one month of Claude 4’s release, it had captured 45% of Anthropic’s user base. The now-cheap previous generation is not harvested for savings. It is abandoned. Every price decline is immediately reinvested in capability, because “good enough” is redefined upward with every release.

This looks like Jevons paradox (cheaper input, more consumption), but the mechanism is subtly different and worth stating precisely. It is not that cheapness induces more usage. It is that the savings are never realized at all, because the buyer changes what they are buying the moment the discount arrives. The 10x price drop applies to a model you no longer use.

There is a rational core to this. Frontier models really do unlock tasks the previous generation failed at, and for those tasks the migration pays for itself. The failure mode is migrating everything, including the 80% of your traffic that a year-old model handles indistinguishably well, at a tenth the price. Frontier-chasing is a portfolio decision that most teams make as a monoculture.

Force 2: Agents are a 1000x multiplier with a 30x error bar

If Force 1 explains why bills did not fall, Force 2 explains why they exploded.

The best empirical work on this is a May 2026 paper from Stanford’s Digital Economy Lab and Microsoft Research, Bai et al., How Do AI Agents Spend Your Money?, and its headline finding deserves to be quoted at every budget meeting: agentic tasks consume on the order of 1,000x more tokens than code chat or single-shot reasoning over the same subject matter.

The mechanism is the agent loop itself. An agent does not answer once. It acts, observes, and re-reads its accumulated context before every next step. A 50-step trajectory does not cost 50 answers. It costs 50 answers, each of which re-ingests everything that came before. Consumption scales with trajectory length times accumulated context, which is quadratic-ish in the length of the task. That is why the paper finds costs dominated not by output tokens (the ones people intuitively think of as “the model working”) but by input tokens: the same context, billed again and again, once per step. Production agent bills routinely run 80 to 95% input.

Two further findings turn this from expensive into unbudgetable. First, identical agents running the identical task varied in cost by up to 30x. Trajectories are stochastic, and a wrong turn early compounds into thousands of extra steps’ worth of re-read context. Second, models systematically underestimate their own token consumption when asked to predict it. You cannot budget an agent workload from the price sheet, and you cannot budget it by asking the model. You can only budget it by metering it.

Notice the compounding with my previous post. The larger the context window you actually use, the more expensive every step of the loop becomes. Where the million-token window is real, it is not just an accuracy risk. It is a cost amplifier bolted to a ratchet.

The math nobody runs before launch

Illustrative numbers, at $3 per million input tokens and $15 per million output. Check them against your own contract.

A support chatbot handling 10,000 conversations a day, averaging 2K input and 500 output tokens per conversation, costs about $135 a day. Comfortable. This is the workload everyone models, because it is the workload that existed when the budget was written.

Now add an agentic workflow: 500 tasks a day, one-twentieth the traffic, each running a 50-step loop whose context grows to an average of 40K input tokens per step. That is 2M input tokens per task, about $6 each, roughly $3,000 a day before output tokens. The agent workload is 5% of your interactions and about 95% of your bill, roughly 450x the cost per interaction. And per the Stanford findings, that $6 figure is a median with a long tail: some runs of the same task will cost $50 or more.

No line on any provider’s pricing page warns you about this. The price per token did not change. The tokens per task changed by three orders of magnitude.

Force 3: The context tax, where grounding gets expensive

The third force is quieter: the cost of making models correct about your data.

Everyone now knows the two ways to ground a model in your documents: retrieve the relevant chunks (RAG) or stuff everything into the long context window. What has been missing is a clean measurement of what the second option costs. A June 2026 study out of Miami University and the University at Buffalo, aptly titled The Token Tax of Epistemic Accuracy, supplied it. On document-grounded QA, long-context stuffing beat semantic RAG on accuracy, 73.1% versus 65.4%. It did so by processing about 248K input tokens per query against RAG’s 8.4K, making the long-context approach 26x more expensive for a 7.7-point accuracy gain. Latency, notably, was a wash.

That is the context tax in one line: accuracy bought with evidence is billed in input tokens, and the exchange rate is terrible at the margin. The last few points of correctness are purchased at a steeply superlinear token price, and because it is all input tokens, it lands on exactly the same side of the bill that agent loops already dominate.

Sometimes the tax is worth paying. For safety-critical or high-value-per-query work, 7.7 points can justify 26x. The failure mode is paying it by default, on every query, because “the window is big enough” quietly became the architecture. That is the same anti-pattern I flagged in the context-window post, now with a price tag attached.

(The industry’s answer to re-billed input tokens is prompt caching, and the honest version of that story, covering write premiums, break-even math, and invalidation semantics that differ wildly between providers, is strange enough that it gets its own post, next in this series.)

If you sign the checks (business leaders)

Change the metric you govern by. Cost-per-token is a vanity number. Demand cost-per-completed-task, per workflow, on a dashboard, before anything agentic scales. If your team cannot produce that number, that fact, not the number itself, is the finding.

Budget agents with variance, not averages. A workload with a documented 30x run-to-run cost spread needs a P95 budget and a per-task kill switch, the way you would cap a cloud autoscaler. “The pilot cost $400 last month” tells you almost nothing about what 100x the volume costs. Agent economics do not extrapolate linearly.

And when a vendor’s price cut arrives, ask your team one question: which workloads are staying on the newly cheap model to bank it? If the answer is “none, we moved everything to the new frontier model,” you did not get a price cut. You got an upgrade you may not have needed everywhere.

If you draw the diagrams (architects)

Treat context as a metered resource with a budget, the way you treat memory or bandwidth. Every agent step that re-reads 40K tokens should be a decision, not a default: prune trajectories, summarize completed sub-tasks, cap tool-output verbosity. The cheapest token remains the one you never send.

Build a model portfolio, not a model choice. Route by task difficulty: frontier models for the 20% of calls that need them, last year’s model, at LLMflation prices, for the rest. Teams that do this well typically report that the majority of traffic runs happily on the cheap tier. The discipline is in the router and the evals that keep it honest, not in the model card.

And revisit RAG versus stuffing as an economic decision per workload, not an ideological one. The 26x tax buys 7.7 points in one published setting. Measure your own exchange rate. Hybrid designs, retrieval to select and long context to integrate, usually dominate both extremes, which is the same conclusion the effective-context data pointed to, arrived at here from the billing side.

If you write the code (senior engineers)

Instrument before you optimize. Per-task token telemetry, broken out by input, output, cached, per step, and per tool call, is a day of work with any modern observability stack, and it converts every claim in this post from anecdote to measurement on your workload. You will find your own 30x outliers within a week, and their trajectories will tell you exactly where the loop went wrong.

Then attack the input side, because that is where the money is: kill runaway trajectories with step and token caps; truncate or summarize tool outputs before they enter context; do not carry full history when a running summary scores the same on your evals. Each of these is boring. Each of them, on an input-dominated bill, is worth more than any prompt cleverness on the output side.

The number nobody prints on the pricing page

The falling-price story is true. That is what makes the paradox durable: everyone’s intuition says costs are handled, right up until the first production agent bill arrives. Per-token price is the number on the box. Cost-per-task is the number that hits the P&L, and it is set by frontier-chasing you did not audit, agent loops you did not meter, and context you did not budget.

The teams that win the next two years of AI economics will not be the ones with the sharpest procurement negotiation on price per million tokens. They will be the ones who know their cost-per-task to the cent, route the boring 80% of their traffic to boring cheap models, and treat every token of context as spend that has to earn its place. Their bills fall with the prices. Everyone else’s bills are the paradox.

This is the second in a series on the operational realities of AI providers that never make the marketing pages. Previously: The Effective Context Window Lie. Next: the prompt-caching economics nobody models, covering write premiums, break-even math, and why dropping one old message can silently reprice your entire conversation.

References & further reading

Agent token consumption

RAG versus long context

Falling inference prices (LLMflation)

Enterprise LLM economics

The counter-view and practitioner accounts

← All posts