Skip to content

AI data market

Agent completion rate vs business value: 15 public runs re-ranked

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Completion rate says how often an agent finishes; business value also depends on what each finish costs, how long it takes, how much human oversight surrounds it and how much damage failures do. Re-ranking 15 public TheAgentCompany runs by estimated model cost per completed task moves seven of them three or more places. Investors should ask for a five-term ledger, not a single agent completion business value metric.

Agent run

Agent completion rate

This page covers agent completion rate vs business value: 15 public runs re-ranked.Illustrative

Key takeaways

  • Across 15 public TheAgentCompany runs, ranking by estimated model cost per completed task moved seven runs three or more places from their completion rank.
  • Two runs on the same scaffold finished 58 and 54 of 175 tasks, yet one cost 2.4 times as much per completed task.
  • In the six highest-completing runs, 61 to 73 percent of failed tasks ended with at least one checkpoint passed: partial work a person must find, finish or undo.
  • Outcome-priced support agents bill on the vendor's own definition of a resolution, so that definition is part of the revenue an investor underwrites.
  • Severity needs its own measure: a side-effect benchmark found unauthorized writes in 12.8 percent of runs that met their goal.

Why is completion rate a weak stand-in for agent business value?

Because it counts finishes and ignores what each finish consumes or breaks. An agent that completes 40 percent of tasks cheaply, quickly and safely can be worth more than one that completes 45 percent while burning compute, waiting on reviewers and now and then doing harm, and a completion leaderboard cannot tell the two apart.

Researchers have made the cost half of this argument for two years. In AI Agents That Matter (arXiv 2407.01502, 2024), Kapoor and colleagues argued that agent evaluations must be cost-controlled, since agents built on stochastic models can raise accuracy just by being called again and again, and they proposed reading results as a Pareto curve of accuracy against inference cost. A November 2025 preprint by Sushant Mehta proposing the CLEAR framework (cost, latency, efficacy, assurance and reliability; arXiv 2511.14136) reports that, for six agents on 300 enterprise tasks, optimizing for accuracy alone produced agents 4.4 to 10.8 times more expensive than cost-aware alternatives with comparable performance.

For an investor the gap matters twice over. It decides whether an agent company's customers keep paying, and it decides which inputs, including the records used to train and test agents, become worth buying. This page measures the gap on published benchmark runs, then sets out the five terms that an agent completion business value metric has to combine before anyone can compare two agents or two vendors.

Fifteen public agent runs, ranked three ways

Ranked by completion rate, by cost per completed task and by steps per completed task, the same 15 runs fall into three different orders, and seven runs move three or more places between the first two rankings. Only the top run holds first place in all three.

TheAgentCompany is a benchmark of 175 professional tasks, each packaged as a Docker image, in which agents browse web services such as GitLab and RocketChat, write and run code and message coworkers played by a language model, with progress scored against checkpoints. Its maintainers keep every leaderboard submission in a public experiments repository: per-task evaluator results plus a README summary giving perfect completions, average steps and average estimated cost in US dollars. This analysis used the 15 run folders that cover all 175 tasks and report both averages, at repository commit f9b1e94 of July 31, 2026. Cost is each run's reported estimate of model spend; the benchmark's own summary script derives it from token counts at list prices it records with an access date, so the figures reflect the prices of their day.

The reversals are large. Gemini 2.0 Flash, run in December 2024, sits seventh on completion with 20 of 175 tasks but third on cost per completed task, because each attempt cost an estimated 0.58 US dollars. Claude 3.5 Sonnet in the same batch finished more than twice as many tasks, 42, yet ranks tenth on cost per completion at 26.42 dollars. Two June 2025 runs on the same OpenHands-Versa scaffold finished 58 tasks with Claude Sonnet 4 and 54 with Claude 3.7 Sonnet, a gap of four tasks, while their cost per completed task differed by a factor of 2.4.

Steps reorder the list again. A step is one action by the agent, a rough proxy for elapsed time and for how much a reviewer must read. Gemini 2.5 Pro and Versa with Claude 3.7 Sonnet finished 53 and 54 tasks, but the Versa run needed 170.9 steps per completed task against 89.9, close to twice as many.

The table shows eight of the 15 runs, with ranks among all 15 and 1 the best. Cost is the benchmark's estimate of model spend, not the full cost of operating an agent.

Fifteen public agent runs, ranked three ways
Run (scaffold, model, folder date)Completed of 175 (rank)Est. model cost per task, USDCost per completed task, USD (rank)Steps per completed task (rank)
TTE-MatrixAgent, DeepSeek V3.2, 2025-11-1075 (1)0.400.93 (1)69.8 (1)
OpenHands-Versa, Claude Sonnet 4, 2025-06-1458 (2)1.634.92 (2)140.2 (5)
OpenHands-Versa, Claude 3.7 Sonnet, 2025-06-1454 (3)3.7011.99 (4)170.9 (7)
OpenHands 0.28.1, Gemini 2.5 Pro, 2025-05-1053 (4)4.2313.97 (6)89.9 (2)
OpenHands 0.14.2, Claude 3.5 Sonnet, 2024-12-1742 (6)6.3426.42 (10)121.5 (4)
OpenHands 0.14.2, Gemini 2.0 Flash, 2024-12-1720 (7)0.585.08 (3)348.7 (10)
OpenHands 0.14.2, Llama 3.3 70B, 2024-12-1712 (10)0.9313.56 (5)305.2 (8)
OpenHands 0.14.2, Qwen2 72B, 2024-12-172 (15)0.2824.50 (9)2,073.8 (15)
A slopegraph of six of 15 public TheAgentCompany runs ranked three ways, by completion rate, cost per completed task and steps per completed task, with Gemini 2.0 Flash rising from 7th to 3rd on cost and Claude 3.5 Sonnet falling from 6th to 10th.
Ranks among 15 public TheAgentCompany runs (1 = best), computed from the experiments repository at commit f9b1e94.

The five-term ledger: what to ask for instead of one number

Net value per attempted task has five terms: verified completion, machine cost, time, human oversight and expected failure loss. A completion rate supplies the first term only, and only when the completion was checked rather than claimed.

Written out, value per attempted task equals the completion rate times the value of one finished task, minus machine cost per attempt, minus oversight cost per attempt, minus expected failure loss per attempt. Time enters twice: as delay, which has a price of its own in work such as support or dispatch, and as the length of trajectory a reviewer has to read.

Each term needs a unit, a denominator and a named source. Request all five per task family rather than as one blended average, because the same agent can be cheap and safe on password resets and slow and risky on refunds.

The ledger names the unit to request for each term, how to measure it and what a completion rate hides:

The five-term ledger: what to ask for instead of one number
TermUnit to requestHow to measure itWhat a completion rate hides
Verified completionShare of attempted tasks whose end state passes an independent checkInspect the system of record or the output, not the agent's report; show claimed and verified side by sideClaimed finishes and partial work counted as success
Machine costModel and tool spend per verified completion, with the price dateTotal spend over a fixed task set divided by verified completionsCheap attempts that rarely succeed, and strong agents that retry expensively
TimeWall-clock minutes and steps per verified completionLog start, finish and every action; report the median and the slow tailFinishes that arrive only after long, costly loops
OversightHuman minutes per attempted task for review, handoff and finishing or undoing workTime-stamped reviewer actions plus escalated and reopened tasksThe person who quietly completes what the agent left half done
Failure lossFrequency times severity of bad outcomes, by severity classSide-effect checks, reversals, refunds, complaints and legal eventsFinished tasks that also did damage

How much human oversight does a completion rate leave out?

Usually most of it. A failed run has to be noticed, taken over and often cleaned up, a successful one may still be reviewed, and none of those minutes appear in a completion rate.

A December 2025 study of agents in production, built on a survey of 306 practitioners and 20 case studies (Pan and colleagues, arXiv 2512.04123), found that 74.2 percent of deployed agents rely on manual, human-in-the-loop evaluation. In the same survey 46.7 percent of agents complete only one to four steps and a further 21.7 percent perform five to ten. Production agents are still kept on a short leash, and the leash is paid for in staff time.

The public runs show what that staff time inherits. In the six highest-completing TheAgentCompany runs, between 61 and 73 percent of failed tasks ended with at least one checkpoint passed: the agent did part of the job and stopped. The benchmark's score gives half its weight to the share of checkpoints passed and half to finishing outright, so it reads partial work as progress. A business reads it as minutes, either saved because a person starts halfway or spent because a person first has to work out what changed.

Companies that report deployments carefully already publish more than a completion figure. Klarna's February 27, 2024 announcement said its support assistant handled 2.3 million conversations in its first month, two-thirds of the company's customer service chats, and set beside that a customer satisfaction score on par with human agents, a 25 percent drop in repeat inquiries and resolution time cut from 11 minutes to under 2. The numbers are self-reported, but the shape is the right one: volume, quality, rework and time reported together.

Can a task be finished and still destroy value?

Yes, and a completion rate scores it as a win. Severity is a separate measurement: an agent that completes the task while writing to the wrong record or messaging the wrong person passes a goal check and fails the business.

A preprint posted on October 25, 2026 (Jiaju Huang and colleagues, arXiv 2610.34167) introduced SABench, 201 email, calendar and ticket tasks whose traces a deterministic oracle checks against an explicit set of authorized writes. Across 11 frontier models, 15.2 percent of runs caused what the authors call collateral damage, 12.8 percent of runs that met their goal still made unauthorized writes, and collateral-damage rates differed 4.7-fold between models with comparable goal rates. Two agents with the same completion rate can carry very different risk.

Severity also arrives through the courts. A case comment published on CanLII describes how British Columbia's Civil Resolution Tribunal, in Moffatt v. Air Canada (2024 BCCRT 149), found the airline liable for negligent misrepresentation after its website chatbot misled a customer about how to request a bereavement fare, and rejected the argument that the chatbot was a separate agent the airline could disown. On any completion count, that conversation ended with an answer.

A usable severity measure sorts failures into classes and attaches a cost to each: a visible miss that a person redoes, an error caught in review, an error that reaches a customer or a ledger, and an event with legal or financial exposure. The rate in the last two classes matters more than the completion rate, and it can only be measured against records of what happened afterwards, such as reversals, refunds, complaints and reopened cases.

When resolution is the billing unit, whose definition counts?

The vendor's, which turns the definition of completion into part of the revenue an investor underwrites. Support-agent vendors that price per outcome have each written down what counts, and the rules differ in ways that move revenue.

Intercom's help center counts a Fin resolution when, after Fin's last answer, the customer either confirms the answer was satisfactory, a confirmed resolution, or leaves without asking for more help, an assumed resolution. A customer who explicitly asks for a human is not counted, and a resolution is deducted if the customer comes back to the same conversation for more help.

Zendesk moved to resolution tiers on May 18, 2026. Its help center separates assisted escalations, where the AI agent contributed before a person finished the job, contained resolutions, where the customer did not ask for further help, and verified resolutions, which an LLM confirms by reading the conversation at its end. Only verified resolutions draw down a customer's allowance.

Read together, the two rules point to the question for any outcome-priced agent company: what share of billed outcomes are assumed rather than confirmed or verified, and what share are later reversed? An assumed resolution is a completion inferred from silence. When silence later turns into a reopened case, a refund or a complaint, revenue rested on a finish the customer never experienced.

Is completion ever the right headline number?

Sometimes. When failures are cheap and visible, oversight per task is about the same for every agent and model spend is small next to the value of a finished task, ranking by completion gives the same answer as the full ledger, and early capability research often sits in that corner.

Cost per completion can mislead as well. Qwen2 72B ranks ninth of 15 on cost per completed task only because its failed attempts were cheap; it finished 2 of 175 tasks and left 173 for people. A ratio of spend to finishes says nothing about the work left behind, which is why oversight and failure loss stand in the ledger as terms of their own.

The stronger objection is about evidence rather than arithmetic. A benchmark can verify end states inside a simulated company, but the oversight and severity terms depend on what happened after a task in a real business: whether the refund was reversed, the ticket reopened or the customer complained. A record collection without those downstream outcomes cannot price severity, and an agent environment without realistic consequences cannot generate them. Outcome-linked operating records are the input that turns the last two terms from estimates into measurements.

What could make the value-ledger argument wrong?

Four developments would weaken the case for a multi-term ledger, and each can be tracked:

  • Model prices could fall far enough that machine cost stops separating agents. The estimates rest on list prices, and the benchmark's summary script records access dates in December 2024 and May 2025 for the prices it codes; if spend per task collapses, completion and oversight decide the ranking and the cost term fades.
  • Oversight could be designed out of narrow workflows. If agents stay in short, structured flows, as most production agents in the 2025 practitioner study did, review time per task may be small and steady enough to leave out of comparisons.
  • Severity checks can miss or be gamed. LLM-verified resolutions and side-effect oracles are only as good as their checks, and SABench's oracle relies on an explicit authorization set that most real businesses never write down.
  • Benchmark runs may not transfer. TheAgentCompany submissions differ in scaffold, date and pricing assumptions, and a simulated software company is not a customer's support queue or a finance team's close.

Diligence questions for agent companies and their data suppliers, SourceX included

Ask every agent company for the five terms on its own task families, with definitions fixed before the numbers arrive, and ask data suppliers whether their records carry the downstream outcomes that let oversight and severity be measured at all.

SourceX sits on the supply side of that second question, and the same questions apply to it. It works with companies that choose to license selected business records to AI developers, taking the work from sourcing and qualification through rights review, contracting, preparation and delivery to payment. Whether a particular company's records hold reversals, reopenings or complaint outcomes is established in review, case by case. SourceX does not publish deal or commercial figures; its company and commercial metrics go to investors directly, and its raise is open only to accredited investors under Rule 506(c), with materials available on request from the investor page.

Questions that separate a measured value claim from a completion headline:

  • What share of your completions pass an independent check, and how many claimed completions failed that check last quarter?
  • What is model and tool spend per verified completion, by task family, and on what price date?
  • What are the median and 90th-percentile human minutes per attempted task, including work finished or undone after a failed run?
  • How do you classify failure severity, and how many events in the two most serious classes did customers report?
  • If you bill per outcome, what share of billed outcomes were assumed rather than confirmed or verified, and what share were reversed within 30 days?
  • Which records of downstream outcomes do you use to measure severity, and do you hold the rights to use them for training and evaluation?

Methods note: files, definitions and limits of the re-ranking

Data: TheAgentCompany experiments repository at commit f9b1e9411fcb554e76509a0e1bc8fe7f8b0e69d6 (July 31, 2026), folder evaluation/1.0.0, cloned and read on 2026-10-27. Inclusion rule: run folders covering all 175 tasks whose README summary reports Perfect Completions, Average Steps and Average Cost (USD); 15 of 19 folders qualify. Excluded: OWL-RolePlay, which reports rounds instead of steps; MUSE, which has no steps or cost line; and two runs on a 41-task subset.

Definitions: completion rate is Perfect Completions divided by 175. Cost per completed task is Average Cost times 175 divided by Perfect Completions, and steps per completed task is the same calculation with Average Steps. A failed task with partial progress is one whose results file shows a final score above zero and below its total. Ranks run across the 15 included runs with 1 the best, tied runs sharing the better rank. Cost is rounded to two decimals and steps to one. For two runs the per-task tables average to slightly different values than the summary lines; the summary lines were used.

Costs are each run's reported estimates of model spend; the main TheAgentCompany repository's evaluation/summarise_results.py derives them from token counts at dated list prices. They leave out infrastructure, tools and people. Runs differ in agent scaffold and date, so they compare systems rather than models, and results were taken as published rather than rerun. Not measured: oversight minutes, failure severity, wall-clock time and the value of a completed task, none of which the benchmark records.

Frequently asked questions

What is a good business value metric for an AI agent?

No single number works. Build net value per attempted task from five terms, verified completion, machine cost, time, human oversight and expected failure loss, and report each per task family. Completion rate is only the first term, and only when an independent check confirmed the finish.

Is cost per completed task better than completion rate?

It adds what each attempt consumes, which completion ignores, but it still ignores the work failures leave behind. In TheAgentCompany's public runs, a model that finished 2 of 175 tasks ranked ninth of 15 on cost per completion because its failures were cheap. Pair the ratio with oversight minutes and failure severity.

How do outcome-priced AI agents define a resolution?

Each vendor writes its own rule. Intercom counts confirmed and assumed resolutions after Fin's last answer and deducts one if the customer returns for more help. Zendesk's tiers, introduced May 18, 2026, separate assisted escalations, contained resolutions and LLM-verified resolutions, and only verified ones draw down the allowance.

Why does failure severity matter if the task was completed?

A goal check can pass while the agent does damage somewhere else. In SABench, 12.8 percent of runs that met their goal still made unauthorized writes, and a British Columbia tribunal held Air Canada liable for its chatbot's misleading answer about a bereavement fare. Severity needs its own classes and its own outcome records.

What should investors ask an AI agent company about its metrics?

Ask for verified completion, spend per verified completion, human minutes per attempted task and severity-classed failure counts, by task family and with definitions fixed in advance. For outcome-priced products, also ask what share of billed outcomes were assumed rather than verified and how many were reversed later.

Sources

  • TheAgentCompany's public experiments repository publishes per-task evaluator results and README summaries (perfect completions, average steps, average estimated cost in USD) for each leaderboard run on its 175 tasks. Source
  • TheAgentCompany's summary script estimates cost from token counts at list prices recorded with access dates and scores tasks with half weight on checkpoint share and half on full completion. Source
  • TheAgentCompany 1.0.0 has 175 tasks, each a Docker image, with LLM-played coworkers; its README describes agents browsing the web, writing code, running programs and communicating with coworkers. Source
  • Kapoor and colleagues argue that agent evaluations must be cost-controlled and propose a Pareto curve of accuracy against inference cost. Source
  • The CLEAR framework preprint reports that optimizing six agents on 300 enterprise tasks for accuracy alone produced agents 4.4 to 10.8 times more expensive than cost-aware alternatives with comparable performance. Source
  • A survey of 306 practitioners and 20 case studies found 74.2 percent of deployed agents rely on manual human-in-the-loop evaluation, 46.7 percent complete 1-4 steps and 21.7 percent 5-10 steps. Source
  • SABench (201 email, calendar and ticket tasks) found collateral damage in 15.2 percent of runs across 11 models, unauthorized writes in 12.8 percent of goal-meeting runs, and a 4.7-fold spread at comparable goal rates. Source
  • Klarna reported that its AI assistant handled 2.3 million conversations, two-thirds of its customer service chats, in its first month, with satisfaction on par with human agents, 25 percent fewer repeat inquiries and resolution time under 2 minutes versus 11. Source
  • A CanLII case comment describes the tribunal in Moffatt v. Air Canada (2024 BCCRT 149) finding the airline liable for negligent misrepresentation by its chatbot and rejecting the separate-agent argument. Source
  • Intercom counts a Fin resolution as confirmed or assumed after Fin's last answer, excludes explicit requests for a human, and deducts a resolution when the customer returns for more help. Source
  • Zendesk's resolution tiers, introduced May 18, 2026, separate assisted escalations, contained resolutions and LLM-verified resolutions; only verified resolutions draw down the allowance. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies