AI uses for records
Agent evaluation cost-success frontier: comparing cost per verified success
By SourceX Editorial · Updated
10 sections
Short answer
Compare agents by cost per verified success: add every cost of running the evaluation set (all attempts, tool calls, verification and the cost of handling failures) and divide by the number of tasks whose completion was verified against evidence. Plot each agent's cost against its verified success rate, discard agents beaten on both, then test which assumption, usually the cost of a failure, flips the choice.
Evaluation split
Agent evaluation cost
Key takeaways
- Cost per attempt and cost per verified success can rank the same agents in opposite orders.
- Verified means the business end state was checked against evidence, not that the agent reported success.
- Failure handling cost is usually the input that decides the comparison, so compute the break-even value explicitly.
- The cost-success frontier keeps only agents no other agent beats on both cost and verified success.
- Show uncertainty on success rates and propagate it into the cost figure before declaring a winner.
- The evaluation set must mirror the real workload mix, or the cheapest agent on paper wins on the easy tasks only.
How do you compare agents by cost per verified completed task?
Divide the full cost of running each agent on the same evaluation set by the number of tasks it completed and that you verified. The full cost includes every attempt and retry, model and tool usage, the verification step itself, and what it costs to deal with each failure. The agent with the lowest result is cheapest per finished task, provided its success rate meets your floor.
The decision rule is short. First, fix a stable evaluation set that mirrors the workload. Second, measure cost and verified success per agent on that set. Third, keep only agents on the cost-success frontier. Fourth, pick the cheapest frontier agent whose lower success bound clears the minimum the business accepts, and check that the choice survives a reasonable range of failure costs.
What changes the answer is rarely the model price. It is how often the agent fails, how expensive each failure is to catch and fix, and whether the evaluation tasks look like the work the agent will actually get.
Write the minimum success floor down before the run. If the business would not deploy any agent below a stated verified success rate, an agent that is cheapest per success but sits below that floor is out, however attractive its cost. Fixing the floor in advance also stops the team from moving it after seeing the results.
Why the cheapest agent per attempt can cost more per finished task
An agent that is cheap to run but fails often hands its failures to someone else, and that cost lands outside the model bill. Cost per attempt hides it; cost per verified success puts it back.
Kapoor and colleagues argue in AI Agents That Matter that accuracy alone cannot identify progress because it can be improved by methods such as retrying, so cost and accuracy should be evaluated together and visualized as a Pareto frontier. The same logic runs in the other direction for buyers: a low per-attempt cost can be bought with a low success rate.
In business workflows the failure cost is concrete. A failed refund agent means a person reopens the case, a failed data-entry agent means a correction and possibly a downstream error, and a failure that slips past verification can cost more than either. Those are the numbers the worksheet needs.
The cost per verified success worksheet
This is the asset. Fill one column per agent from the same evaluation run. Values in the example column are illustrative and expressed in normalized cost units, where one unit is whatever internal cost basis your team uses; the method does not depend on currency.
- Total cost = N × a × (c + v) + F × r + escaped failures × e.
- Verified success rate = S ÷ N.
- Cost per verified success = total cost ÷ S.
- Report both per-attempt cost (c + v) and cost per verified success, so readers see why the ranking changes.
| Input | Definition | Where it is measured | Illustrative value |
|---|---|---|---|
| Tasks (N) | Distinct tasks in the evaluation set, weighted to the workload mix | Evaluation set manifest | 200 |
| Attempts per task (a) | Runs per task the deployment allows, including retries | Run logs | 1 |
| Run cost per attempt (c) | Model tokens, tool and API calls and compute for one attempt | Usage records per run, priced at your rates | 1.0 units |
| Verification cost per attempt (v) | Automated checks plus any human review needed to confirm the end state | Grader logs and reviewer time | 0.5 units |
| Verified successes (S) | Tasks whose business end state was confirmed against evidence | Grader output with evidence links | 110 |
| Caught failures (F) | Tasks verified as not done, handled by a fallback | Grader output | 90 |
| Failure handling cost (r) | Cost to finish or undo a failed task, from the fallback process | Human baseline or rework records | 6.0 units |
| Escaped failure weight (e) | Expected cost of failures verification missed, by severity | Audit sample of verified successes | 0 in this example |
What counts as a verified success?
A success is verified when the required end state was checked against evidence the agent could not write itself, such as the record in the system of record, a reconciled ledger entry or a document a reviewer confirmed. The agent saying it finished is not verification.
Decide how retries count before running anything. tau-bench's authors contrast pass@k, where a task counts as solved if any one of k trials succeeds, with pass^k, the chance that all k trials succeed, and argue pass^k captures reliability. If production will run the agent once per task, measure single-attempt success. If production can retry under a verifier, count the cost of every attempt and credit success only once.
Escaped failures belong in the worksheet too. Audit a sample of verified successes by hand; if some were not actually done, estimate the escape rate and weight it by severity, because a failure that reaches a customer usually costs more than one caught in review.
Partial completions need a rule too. A task with three required outcomes where the agent achieves two is either a failure, a success with a separate partial count, or a fraction of a success. Pick one, apply it to every agent and state it next to the result, because the choice can shift the success rate by more than the difference between the agents.
Which usage and pricing inputs move the number?
Measure usage per attempt from logs rather than estimating it from averages, because long-tail attempts drive cost. Token counts, tool calls and wall-clock time per attempt are the core measurements.
The OpenTelemetry GenAI semantic conventions define attributes for this, including gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus separate counts for input tokens read from or written to a provider-managed cache. Recording these per run makes the worksheet reproducible.
Pricing structure matters as much as rates. Anthropic's Claude API pricing, for example, is quoted per million tokens with input and output priced separately, cache writes and cache reads priced differently from base input, and batch processing discounted. Two agents with the same token totals can therefore cost different amounts if one makes heavy use of cached context or of output. Apply your actual contracted rates to logged usage and record the pricing date, since rates change.
Count tool costs as carefully as model costs. An agent that calls a paid enrichment service, a search API or a slow internal system on every step can be cheap in tokens and expensive overall. Record each external call with its count and unit cost per attempt, and include the verifier's own model calls if it uses one, since verification can cost as much as the attempt it checks.
Drawing the cost-success frontier
Plot each agent configuration as a point, with cost per task on one axis and verified success rate on the other. An agent is on the frontier when no other agent is both cheaper and more successful; every other point is dominated and can be dropped.
Research leaderboards now publish this view. The Holistic Agent Leaderboard plots accuracy against cost for each benchmark and computes the frontier automatically, which lets readers choose under a budget rather than by accuracy alone. Your internal version should use verified success on your own tasks, not a public benchmark score.
The frontier narrows the choice; it does not make it. Among frontier agents, apply the business floor on success and then choose by cost per verified success. Include configurations of the same model, such as different retry limits or reasoning settings, as separate points, because they often trace the frontier more finely than different models do.
Plot the interval for each point, not only the point. Two agents whose success intervals overlap heavily are not meaningfully different on that axis, and a frontier drawn from point estimates can show one dominating the other when the data cannot support it. Mark such pairs as ties and decide between them on cost, latency or operational fit.
Worked scenario: a cheap agent against a careful one
This scenario is illustrative; all values are invented normalized units and counts. Both agents run once on the same 200 tasks. Agent A costs 1.0 unit per attempt and verifies 110 successes. Agent B costs 2.5 units per attempt and verifies 170 successes. Verification costs 0.5 units per attempt for both, and each caught failure costs 6.0 units to hand off and finish.
Per attempt, A looks cheaper at 1.5 units against 3.0. Per verified success, A's total of 300 units for attempts and verification plus 540 for 90 failures comes to 840 units over 110 successes, about 7.6 units each. B's 600 units plus 180 for 30 failures comes to 780 units over 170 successes, about 4.6 units each.
The cheaper agent is the more expensive way to get work done in this scenario. If failure handling cost nothing, the ranking would reverse: A at about 2.7 units per verified success against B at about 3.5.
The scenario also shows why reporting only per-attempt cost misleads procurement. A budget built on A's lower attempt price would be exceeded once the fallback team's time is counted, and the overrun would appear in a different department's numbers.
Break-even failure cost and the error bars around it
In most comparisons the failure handling cost decides the winner, so compute the value at which the two agents tie. In the scenario, A's cost per verified success is (1.5 + 0.45r) ÷ 0.55 and B's is (3.0 + 0.15r) ÷ 0.85. They are equal when r is 1.25 units; above that, B is cheaper per finished task. If the business is confident failures cost more than that, the decision is robust to the exact figure.
Success rates carry sampling error, and that error flows into cost. A Wilson score interval, available in statsmodels' proportion_confint, gives about 48% to 62% for A's 110 of 200 and about 79% to 89% for B's 170 of 200 at 95% confidence. Plugging the ends of those ranges into the formula with r at 6.0 brackets A at roughly 6.2 to 9.6 units and B at roughly 4.1 to 5.3; this is a simple bracketing for illustration, not a formal interval for the ratio.
For a proper interval on the difference, resample tasks. SciPy's bootstrap function computes a bootstrap confidence interval for a statistic and has a paired option that resamples the same task indices for both agents, which respects the fact that they ran on identical tasks. Report the interval, the break-even failure cost and the workload weights alongside the headline number.
Is the evaluation set missing the tasks where the cheap agent fails?
Check coverage before trusting the comparison, because a missing task family can reverse it. A cheap agent that handles routine cases well and fails on exceptions will look strong on a set that under-samples exceptions.
Build a simple coverage matrix with task families down one side and failure types across the other, and count evaluation tasks in each cell. Weight the cost calculation by the share of each family in production volume, not by its share of the evaluation set. Where a high-severity cell is empty, say so in the result and do not extrapolate.
Rerun the worksheet when the workload mix, the success criteria or an agent's model version changes. The comparison describes one workload at one time, not the agents in general.
Failure handling cost also varies by task family. A failed password reset is cheap to redo; a failed vendor payment change may need an investigation. Where the data allows, estimate the handling cost per family from rework or human baseline records and apply it family by family, rather than using one average for every failure.
Building the evaluation set from your own workload
The worksheet is only as good as the tasks it runs on, and the most reliable tasks come from real completed work with recorded outcomes, which gives both the evidence for verification and the history to estimate failure handling cost.
AI buyers can ask SourceX to source records like these from U.S. businesses, since nothing is held in stock; the buyer shares the scope, volume, language, format, timing and licensing needs, which can include the task families and outcome evidence an evaluation set like this requires.
Frequently asked questions
What is cost per verified success for an AI agent?
It is the total cost of running an agent over an evaluation set, including retries, verification and failure handling, divided by the number of tasks whose completion was checked against evidence. It measures the cost of finished work rather than the cost of attempts.
What is a cost-success frontier?
It is the set of agents for which no other agent is both cheaper and more successful on the same evaluation. Agents off the frontier are dominated and can be dropped before choosing among the rest.
Should retries be included in agent cost?
Yes, if the deployment will retry. Count the cost of every attempt and credit a task as one success at most; if production runs the agent once, measure single-attempt success instead.
How large should the evaluation set be for a cost comparison?
Large enough that the success rate intervals do not overlap in a way that changes the decision. Compute Wilson intervals or a paired bootstrap and check whether the cheaper agent remains cheaper across their range.
Why can a cheaper model end up costing more?
Because its failures still need to be caught and fixed, often by people. When failure handling costs more than the break-even value, an agent with a higher price per attempt but fewer failures costs less per finished task.
Sources
- AI Agents That Matter argues accuracy alone can be inflated by retrying, so agents should be evaluated on cost and accuracy jointly, visualized as a Pareto frontier. Source
- The Holistic Agent Leaderboard plots accuracy against cost and computes the frontier automatically for each benchmark. Source
- tau-bench contrasts pass@k (any one of k trials succeeds) with pass^k (all k trials succeed) as a reliability measure. Source
- Claude API pricing is quoted per million tokens, with input and output priced separately, cache writes and reads priced differently and batch processing discounted. Source
- OpenTelemetry GenAI conventions define per-call input, output, cache read and cache write token count attributes. Source
- statsmodels' proportion_confint provides a Wilson score interval for a success proportion. Source
- SciPy's bootstrap computes a confidence interval for a statistic and can resample paired samples by shared indices. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.