Skip to content

Leadership and readiness

AI evaluation cost in capability comparisons: same score, 10x tokens

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Capability charts need a cost axis, because equal scores can hide a tenfold difference in spend. In METR's public time-horizon runs, GPT-5.3-Codex and Claude Opus 4.5 succeed at nearly the same weighted rate, 0.745 and 0.730, yet GPT-5.3-Codex's median run uses 10.4 times as many tokens. Cap every agent at 100,000 tokens a run and 28 percent of model pairs change order.

Evaluation split

AI evaluation cost

This page covers AI evaluation cost in capability comparisons: same score, 10x tokens.Illustrative

Key takeaways

  • A capability chart without a cost axis hides the budget behind each point: tokens, samples, limits and human oversight.
  • In METR's public runs, two pairs of agents with nearly equal weighted success differ about tenfold in median tokens per run.
  • Capping every agent at 100,000 tokens a run reorders 43 of 153 model pairs; GPT-5.2 drops from second to thirteenth.
  • Even uncapped scores carry a budget: four agents had 97 to 125 runs stopped at METR's usage limits.
  • Oversight is paid in minutes: the strongest agent clears 85.1 percent of tasks but only 56.7 percent of the suite's expert minutes.
  • Leaderboard token columns can omit hidden reasoning, which OpenAI bills as output tokens, so ask what each budget unit counts.

What does a capability chart leave out?

It leaves out what each point cost to produce: the tokens and samples the system spent, the time and usage limits the evaluator set, and the human work that checks or finishes what the system could not. Two models on the same line of a chart can sit there at very different prices.

That matters for an AI evaluation cost capability comparison because the investable quantity is task success per unit of budget, not task success alone. A product's margins depend on what a completed task costs, and a data company's claim that its records lifted a score only holds if the comparison ran at the same budget.

A widely cited capability chart shows the problem. METR's time-horizon work, whose repository summarizes the key finding as agent time horizons doubling about every 7 months, plots each model's result against its release date. METR also publishes the run file behind that chart, including a token count for every run, which makes it possible to put the missing axis back. This page does that, then turns the result into a worksheet.

Same success, ten times the tokens: METR's runs with the cost column restored

The run file shows models with almost the same success rate spending an order of magnitude more or less to get it. METR's Time Horizon 1.1 file holds 24,008 runs on 228 tasks from 79 task families, and 18 of its 20 agents have a token count on every run. Success here is METR's diversity-weighted success rate over the whole suite, not its fitted time horizon.

GPT-5.3-Codex reaches 0.745 and Claude Opus 4.5 reaches 0.730, but the median GPT-5.3-Codex run uses 138,060 tokens against 13,310: a ratio of 10.4. GPT-5.1-Codex-Max and Gemini 3 Pro make the same pattern at 0.708 and 0.710, with medians of 181,916 and 17,672 tokens, a ratio of 10.3. On a chart of success alone each pair looks like a tie.

Tokens are a budget proxy, not a price: models charge different rates per token, and the file does not split its count into input, output and reasoning tokens. Yet a tenfold gap in tokens at equal success is too large for pricing differences to dismiss without checking. The eight strongest agents in the file:

Same success, ten times the tokens: METR's runs with the cost column restored
Agent (METR run file)Weighted success, no capMedian tokens per runSuccess at 100,000-token cap (rank of 18)Runs stopped at a usage limit
Claude Opus 4.60.78919,9750.569 (1)18 of 1,391
GPT-5.20.753206,3400.279 (13)97 of 961
GPT-5.3-Codex0.745138,0600.346 (10)105 of 1,602
Claude Opus 4.50.73013,3100.567 (2)11 of 1,044
Gemini 3 Pro0.71017,6720.526 (3)12 of 1,044
GPT-5.1-Codex-Max0.708181,9160.293 (12)125 of 1,369
GPT-50.69463,1310.377 (8)51 of 1,056
o30.636150,1850.273 (17)112 of 1,044

Do matched token budgets reorder the leaderboard?

Yes, and the tighter the budget, the more it reorders. METR's own analysis code includes a success-at-cost view in which any run above a usage cap counts as a failure; applying that rule at a fixed token cap for every agent gives a matched-budget leaderboard.

At 100,000 tokens a run, 43 of the 153 model pairs, 28.1 percent, change order relative to the uncapped ranking. GPT-5.2 falls from second to thirteenth, o3 from eighth to seventeenth, and Claude 4 Opus rises from tenth to fourth. At 10,000 tokens more than half the pairs reorder (52.3 percent); at 1 million it is 11.8 percent, and Gemini 3 Pro leads; at 10 million only 2 pairs, 1.3 percent, differ from the uncapped order.

A dominance test gives the same message without choosing a cap. Plotting weighted success against median tokens per run, 13 of the 18 agents are dominated: another agent succeeds at least as often with a smaller median run. The frontier holds GPT-4 1106, GPT-4o, Claude 3.5 Sonnet (New), Claude Opus 4.5 and Claude Opus 4.6.

None of this says one lab's models are better. It says that the order of a capability chart is a function of the budget each run was allowed, and a chart that does not state the budget has made that choice silently.

Line chart of weighted success against a per-run token cap from 10,000 to 10 million tokens for four agents in METR's public runs: Claude Opus 4.5 and Gemini 3 Pro rise early while GPT-5.3-Codex and GPT-5.1-Codex-Max climb late, so the pairs that tie without a cap are far apart at 100,000 tokens.
Source: SourceX recount of METR's Time Horizon 1.1 run file (eval-analysis-public, commit 52cb829). Runs above a cap count as failures. Tokens are a budget proxy, not a price.

The uncapped score has a budget too

Every published score was produced under some limit, so an uncapped number is really a number at the evaluator's cap. METR describes Time Horizon 1.1 runs as working up to a token and time limit, and says it checks afterward whether a model had enough token budget. In the run file, GPT-5.1-Codex-Max, o3, GPT-5.3-Codex and GPT-5.2 each have between 97 and 125 runs stopped at a usage limit, against 11 to 18 for Claude Opus 4.5, Gemini 3 Pro and Claude Opus 4.6.

Raising the limit can buy more score. ARC Prize reported that a preview version of OpenAI's o3 reached 75.7 percent on the ARC-AGI semi-private evaluation within its public leaderboard's cost limit, and 87.5 percent in a high-compute configuration using 172 times the compute. Both are true results; they answer different questions about cost.

The research community has argued this for agents since 2024. In AI Agents That Matter (arXiv 2407.01502), Kapoor and colleagues at Princeton argued that agent benchmarks focused narrowly on accuracy reward needlessly costly systems, proposed jointly optimizing accuracy and cost and plotting results as a Pareto curve, and showed simple baseline agents beating many complex ones on HumanEval at a fraction of the cost.

What the success rate leaves for people: the oversight budget

Most of the expert time in the task suite, because the failures cluster in long tasks. On the 221 tasks every agent attempted, the 67 that take a qualified person an hour or more hold 94.9 percent of the suite's expert minutes, while the median task takes 12 minutes.

Weighting each task by those human minutes changes the picture. Claude Opus 4.6 succeeds on 85.1 percent of tasks on average but on only 56.7 percent of the expert minutes; GPT-5.2 succeeds on 83.8 percent of tasks and 48.2 percent of minutes; o3 on 74.8 percent of tasks and 23.1 percent of minutes. Even for the strongest agent, failed runs leave an expected 12,656 of the suite's 29,202 expert minutes undone.

That remainder is the floor of the oversight budget: someone must notice the failure, then finish or redo the work. Checking the successes costs more on top, and the run file cannot measure it. A cost-aware comparison therefore needs two budgets per point, machine spend and human minutes, and should compare models where both are matched.

Could the budget, not the model or the data, explain the gain?

Often it can. Benchmark gains may reflect changed models, prompts or inference budgets rather than data, and a budget change is the easiest to miss because many leaderboards do not report it. A company that says its data improved a model should show the score moving with the data as the only change, at the same token cap, sampling rule and time limit.

Reported cost columns can mislead as well. HELM Capabilities, a public multi-model leaderboard, lists GPT-5's mean output on MMLU-Pro as 5.03 tokens per question with an observed inference time of 18.67 seconds, and o3's as 5.04 tokens with 17.31 seconds; 13 of its 68 models show zero or missing output tokens. OpenAI's reasoning guide explains the gap: reasoning tokens are not visible through the API but still occupy the context window and are billed as output tokens. A token column that counts only visible text understates the budget of exactly the models that think longest.

The diligence rule follows: ask what the budget unit counts, whether hidden reasoning is included, and whether the compared systems ran under the same cap. Without those three answers a capability gap is not yet attributable to anything.

Benchmark underwriting worksheet for cost-aware capability claims

Run the worksheet on every capability chart or score table in a company's materials, top to bottom; the first failing line limits what the chart can support. It was built for this article from the METR recount and the sources above.

The replication checklist comes first, for the analyst rebuilding a cost-aware chart from run files, followed by the worksheet lines:

  • Get the run-level file: one row per attempt with task, model, outcome, token count, sample count, time used and any limit hit.
  • Recompute the headline success rate from runs with the published weights, and confirm it matches.
  • Apply two or three fixed token caps, counting runs above each cap as failures, and report the ranking at each.
  • List runs stopped by usage or time limits for each model, and rerun a sample with a higher limit if counts differ widely.
  • Weight task success by the human minutes each task takes, and report the minutes left in failed tasks.
  • Convert tokens to spend with each provider's published prices on a stated date, keeping reasoning tokens in the count.
Benchmark underwriting worksheet for cost-aware capability claims
LineAsk forPasses whenFails when
1. Budget disclosedToken, sample and time limits for every plotted pointEach point states its limitsOnly scores are shown
2. Budget unitWhat the token count includes: input, output, reasoning, cachedHidden reasoning tokens are countedOnly visible output is counted
3. Matched comparisonScores for all compared systems at one or more shared capsThe ranking holds at the caps that matter for the productThe lead exists only without a cap
4. Sampling ruleAttempts per task and how a final answer was chosenOne attempt, or a stated selection rule applied to everyoneBest of many attempts for one system only
5. Limit hitsRuns stopped by usage or time limits per systemLimit hits are rare or similar across systemsOne system hits the limit far more often
6. Oversight minutesHuman minutes per task and the minutes left in failed tasksHuman time is reported beside successSuccess is shown with no human cost
7. Data attributionA run where only the data changed, at a fixed budgetThe gain survives at the same cap and promptData, prompt and budget changed together

What could make the cost-axis argument wrong

The thesis is that cost-aware comparisons will separate real capability from spend, so evidence that holds at matched budgets deserves the premium. Five things could weaken it:

  • Token prices could fall far enough that a tenfold token gap stops mattering for most products, leaving success alone as the useful measure.
  • Tokens are not cost. Prices differ by model and by input or output, caching changes the bill, and the run file's counts do not separate these, so a token comparison can overstate or understate a spending gap.
  • Post-hoc caps are approximations. Agents in the file did not know the cap, and an agent told about a budget might adapt and do better than a cap applied afterward suggests.
  • The suite is near its ceiling. METR's Frontier Risk Report (May 2026) said the most capable agents essentially saturated Time Horizon 1.1, so differences among the leaders carry growing uncertainty.
  • Scaffolds differ. METR ran agents under several agent frameworks, and some of a token gap may belong to the framework rather than the model.

Questions for model and data companies, SourceX included

Ask every company that shows a capability chart for the budget behind each point, and ask data suppliers whether their records support evaluation at fixed budgets. The same list applies to SourceX.

SourceX sits on the data side of the question. It helps companies license operational records they select to AI developers, handling qualification and rights review, the agreement, preparation, delivery and payment. Records of completed work with known durations and outcomes are one way to build tasks whose human minutes are measured rather than estimated. SourceX has published no evaluation or benchmark results, and this article reports none.

SourceX keeps its company and commercial figures private and shares them with investors directly. Its raise is open to accredited investors only, under Rule 506(c), and the investors page takes materials requests with any of these questions attached:

  • What token, sample and time limits produced each point on your capability chart?
  • Does your token count include hidden reasoning tokens, and what did each run cost at published prices?
  • How do the compared systems rank at a shared token cap that matches your product's budget?
  • How many runs hit a usage or time limit, for your system and for each comparison system?
  • How many human minutes do the tasks your system fails represent, and who covers them in production?
  • Which run isolates the effect of your data at a fixed budget?

Methods note: recounting METR's time-horizon run file

Data: reports/time-horizon-1-1/data/raw/runs.jsonl and data/external/release_dates.yaml in METR's public eval-analysis-public repository at commit 52cb829 (March 5, 2026), read from a blobless clone on 2026-10-27 without running repository code. HELM figures come from groups/core_scenarios.json in the v1.15.0 Capabilities release (dated 2025-11-24), downloaded the same day. Scripts: metr_budget_recount.py and helm_token_column_check.py, kept with this article's working files for reviewers.

Rules as the scripts apply them: agents are all aliases except human, and budget results use the 18 agents with a token count on every run; task success is the mean of score_binarized over an agent's runs of a task; weighted success sums task success times the task's summed invsqrt_task_weight; at a cap, a run counts as a success only if it succeeded and used no more tokens than the cap; a pair is reordered when its order at the cap is opposite to its uncapped order, with ties left out; an agent is dominated when another has success at least as high and median tokens per run at most as high, one of them strictly; long tasks have human_minutes of 60 or more; minute shares use the 221 tasks every kept agent attempted; a median token count that falls on a half token is rounded up. No random sampling was used, so no seed applies.

Not measured: dollar cost, the split of tokens into input, output and reasoning, agent behavior under a cap known in advance, time spent checking successful runs, METR's fitted time horizons and any SourceX data. Agent names follow METR's aliases, without its (Inspect) suffix.

Frequently asked questions

Why should evaluation cost appear on an AI capability chart?

Because the same score can be bought at very different prices. In METR's public runs, two pairs of agents with nearly equal success differ about tenfold in median tokens per run, and capping all agents at 100,000 tokens changes the order of 28 percent of model pairs.

What is a matched-budget comparison of AI models?

It scores every system under the same limits, such as a token cap per task, one attempt per task and a shared time limit, and counts any run that exceeds the limit as a failure. It answers which system does more within a fixed spend, which is the question a product budget asks.

Are tokens a good measure of inference cost?

They are a usable proxy with known gaps. Prices differ by model and by input or output, caching lowers some costs, and some leaderboards count only visible output; OpenAI's documentation says hidden reasoning tokens are still billed as output tokens. Convert tokens to spend with dated price lists before comparing.

How does human oversight change an AI evaluation result?

A success rate counts tasks, while oversight is paid in minutes. In METR's suite the strongest agent succeeds on 85.1 percent of tasks but only 56.7 percent of the expert minutes, because failures sit in long tasks that a person must then finish or redo.

Has SourceX published benchmark or evaluation results?

No. SourceX has published no benchmark or evaluation results, and its company and commercial figures are not public. Investors receive them directly through the materials request on the investors page.

Sources

  • METR's Time Horizon 1.1 run file records, for 24,008 runs on 228 tasks, the agent, task, binarized score, human minutes, diversity weight, token count and any fatal error such as a usage-limit stop. Source
  • METR's eval-analysis-public README describes the time-horizon method and states that AI agent time horizons have been doubling approximately every 7 months. Source
  • METR's success-at-cost code sets the score of every run above a usage cap to zero before averaging by task and weighting tasks. Source
  • METR's Time Horizon 1.1 post describes a 228-task suite whose runs work up to a token and time limit, with a later check of whether each model had sufficient token budget. Source
  • METR's Frontier Risk Report (May 2026) says the most capable agents essentially saturated the Time Horizon 1.1 benchmark. Source
  • ARC Prize reported that a preview version of OpenAI's o3 scored 75.7 percent on the ARC-AGI semi-private evaluation within its public leaderboard cost limit and 87.5 percent in a high-compute (172x) configuration. Source
  • Kapoor and colleagues argued that accuracy-only agent benchmarks reward needlessly costly agents, proposed jointly optimizing accuracy and cost with Pareto curves, and found simple baselines that beat complex agents on HumanEval at lower cost. Source
  • HELM Capabilities v1.15.0 lists GPT-5's mean MMLU-Pro output at 5.03 tokens with 18.67 seconds observed inference time and o3's at 5.04 tokens with 17.31 seconds; 13 of 68 models show zero or missing output tokens. Source
  • HELM's schema defines output tokens as the actual number of output tokens and observed inference runtime as dependent on the particular deployment. Source
  • OpenAI's reasoning guide says reasoning tokens are not visible through the API but still occupy the context window and are billed as output tokens. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies