AI Models: Who Gets the Job Done at a Lower Cost?
The cheapest model per token is often the most expensive one in production. Here is the real math behind AI cost — retry taxes, hidden billing traps, and the one metric that actually matters.
Every model you can call — GPT, Gemini, Claude, Qwen, DeepSeek — is sold on cost per token. Every model is paid for on cost per successful task. Those are not the same number, and the gap between them is where AI budgets quietly disappear.
The reason is simple: cost per token is an infrastructure metric. Cost per successful task is a business metric[reference:4]. The first appears on your API invoice. The second appears on your payroll, in your product delays, and in the engineering hours spent fixing what the model got wrong.
This article explains why the cheapest model on the pricing page is often the most expensive one in production — and how to calculate the number that actually matters.
The Illusion of the Per-Token Price
When a new model is released, the first number everyone looks at is price per million tokens. It is simple, comparable, and presented as if it settles the question of cost efficiency.
It does not.
A model that charges $0.15 per million tokens may consume four times the context of a model that charges $0.60. A model that looks cheap may need three attempts to produce a usable output. A model that appears expensive may succeed on the first try, every time.
Two models can have very different behavior even when they appear close on quality benchmarks. One may produce short, direct answers. Another may generate long chains of reasoning, retry subproblems, or spiral on complex tasks. In engineering and agentic workflows — where models iterate, plan, invoke tools, and handle multi-step tasks — token consumption can vary dramatically[reference:5].
The per-token price tells you what a provider charges for one unit of text. It tells you nothing about how many units a model will burn to finish your work.
The Real Cost of One Attempt: TCO
The total cost of ownership for a single attempt at a task has three components:
TCO = C_token + P(fail) × L
- C_token — the token cost for one attempt.
- P(fail) — the probability that the output requires human intervention (won’t compile, breaks a schema, hallucinates a value).
- L — the fully loaded human repair cost (hourly rate × repair time).
A cheap model minimizes the first term and quietly inflates the second. The first appears on your API invoice. The second appears on your payroll.
This is not a theoretical exercise. Consider two models working on a hard agentic request:
| Budget Model | Frontier Model | |
|---|---|---|
| Token cost per attempt | $0.08 | $1.04 |
| Failure rate | 45% | 8% |
| Human repair cost | $75 | $75 |
| Expected total cost | $33.83 | $7.04 |
Despite costing 13× more per token, the frontier model is 4.8× cheaper per completed task. The savings did not disappear. They moved from the API invoice to the payroll line.
The Retry Tax: What Failure Actually Costs
The retry tax is the extra spend created when a model needs multiple attempts to finish one task. It is the most practical way to express hidden inefficiency in agentic workflows[reference:6].
The math is straightforward. If a model succeeds with probability p, the expected number of attempts to reach one success is 1/p.
| Success Rate | Expected Attempts | Retry Tax vs. 91% Baseline |
|---|---|---|
| 91% | ~1.10 | 1.0x (baseline) |
| 80% | 1.25 | +14% |
| 66% | 1.52 | +38% |
| 50% | 2.00 | ~1.8x |
| 39% | 2.56 | ~2.3x |
| 20% | 5.00 | ~4.5x |
Note the shape of the curve. Dropping from 91% to 80% only adds a 14% tax. But the curve rises steeply once you fall below 50%. A model that succeeds 20% of the time needs about 4.5× the tool calls to produce the same outcome as a model that succeeds 91% of the time.
And this formula is a lower bound. Every retry carries the full error response back into context, adds an LLM turn for cause analysis, and inflates the conversation history for every subsequent turn[reference:7]. The real retry tax compounds across token volume, call count, and wall-clock time.
Six Billing Traps the Rate Card Does Not Show
Even if you account for retries and human repair, the per-token rate is still not the full story. Six billing mechanisms sit between the rate card and your invoice, and each one can move the bill further than switching vendors would[reference:8].
1. Cache asymmetry. Caching is sold as a discount, but only half of it is. On Anthropic models, cache writes cost 1.25× the base input rate and cache reads cost 0.1×. If you write to the cache and never read from it, you have simply paid 25% extra. OpenAI adopted the same shape for GPT-5.6 and later[reference:9].
2. Peak-hour billing. Some providers charge different rates for the same workload at different times of day. A cost model built before a pricing change can be wrong by a multiplier, not a percentage[reference:10].
3. Context cliffs. Some models have a hard threshold after which every token in the request bills at a higher rate. A 201K-token prompt does not cost slightly more than a 199K one. It costs twice as much[reference:11].
4. Uncounted reasoning tokens. Some models generate thinking tokens before answering. Those tokens bill as output, but they never appear in the visible response. A short answer to a hard question can still produce a large output bill[reference:12].
5. Platform fees. Aggregators may charge a percentage on credit top-ups, plus a fixed per-transaction component. The fee is not a markup on tokens, but it is still a cost that does not appear on the vendor’s rate card.
6. Subscription math. A flat monthly plan converts a variable cost into a fixed one — but it also converts a spending problem into an availability problem. When the quota drains, you stop working rather than paying more.
What to Measure Instead
The metric that matters is cost per successful task. It is the total spend across every attempt — including retries, failed tool calls, abandoned runs, and human review — divided by the number of tasks that actually completed successfully[reference:13].
Arize and Fireworks benchmarked 10 models across 2,400 agent runs. The results were blunt: the labels “open” and “closed,” “cheap” and “expensive” were the wrong axis. What separated the models was cost per successful task. One open model with a 33% pass rate cost $0.054 per success with a 3.0× retry tax, while a frontier model with a 67% pass rate cost $0.636 per success but carried only a 1.5× retry tax[reference:14]. The cheapest sticker price did not produce the cheapest completed work.
For production teams, the evaluation framework should include:
- Task completion cost — total spend to get a successful result.
- Success rate — how reliably the model solves the task.
- Token efficiency — how much context the model consumes along the way.
- Failure behavior — whether the model loops, overthinks, or requires retries.
- Operational fit — how well it works inside your real prompts, tools, and workflows.
Route on Complexity, Not on Price
The optimal architecture is not to pick one model. It is to route each request by complexity.
Route boilerplate work — CRUD operations, SQL generation, extraction, classification — to the cheapest model, where failure probability and repair cost both remain low.
Route ambiguous integrations, system design, architecture, planning, and difficult reasoning to the frontier model, where preventing even one expensive engineering mistake easily pays for the additional token cost.
Using an expensive model for trivial work wastes API budget. Using a cheap model for architectural work wastes engineering time. Only one of those appears on your cloud invoice. The other appears in missed deadlines, rework, and the quiet erosion of trust in the system.
The Bottom Line
Cost per token is what you are billed. Cost per successful task is what you actually pay.
The cheapest model on the pricing page is not the cheapest model in production. The model with the lowest per-token rate is not the model that finishes the job at the lowest total cost. The number that matters is the one that includes retries, hidden billing mechanics, and the human labor required to turn a model’s output into work that is actually done.
At NeuralRating, we are building our cost benchmarks around cost per successful task — not price per token. Because a benchmark that ignores the cost of failure is not measuring what you pay. It is measuring what the provider wants you to see.
If you want to follow along, you can join the waitlist on our homepage, read more about our approach, or explore other insights.
Frequently asked questions
Why is cost per token misleading?
Cost per token tells you what a provider charges for one unit of text. It does not tell you how many units a model will consume, how often it will fail and need retries, or how much human labor its errors will create. A model that is cheap per token but needs three attempts and a human fix can cost far more than a model that succeeds on the first try.
What is the retry tax in AI?
Retry tax is the extra cost created when a model needs multiple attempts to complete one task. If a model succeeds 80% of the time, the expected number of attempts is 1.25 — meaning you pay 14% more on average just from retries. Below 50% success rate, the tax rises steeply.
What are hidden costs in LLM API billing?
Beyond the per-token rate, providers charge for cache writes (often 1.25x the input rate), apply context cliffs that can double the entire request past a token threshold, and may not include reasoning tokens in the visible output count. These mechanisms can move your bill further than switching vendors would.
What is the right metric to compare AI model costs?
Cost per successful task. This is the total spend across every attempt — including retries, failed tool calls, and abandoned runs — divided by the number of tasks that actually completed successfully. It is the only metric that reflects what you pay for work that gets done.
Should I use the cheapest model for everything?
No. The optimal strategy is to route by complexity. Use cheap models for boilerplate work where failure is cheap and rare (extraction, classification, simple generation), and use frontier models for tasks where a single mistake is expensive to repair (architecture, planning, complex reasoning).