Skip to content

Deal economics

Reliable agent workflow data cost: why upkeep outgrows the build

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Reliable agent workflow data cost is mostly evaluation, and most evaluation cost recurs. Building tasks and expert answers is paid once; repeated runs to measure consistency, re-review when rules change, re-runs when models change and fresh cases to beat contamination keep arriving. In an illustrative model, upkeep passes the build during year two. Value suppliers on the recurring lines, not on a one-off archive.

Agent run

Reliable agent workflow data

This page covers reliable agent workflow data cost: why upkeep outgrows the build.Illustrative

Key takeaways

  • Reliability is measured over repeated runs, so each task is executed several times for every model, prompt or tool change.
  • The Holistic Agent Leaderboard ran 21,730 agent rollouts across 9 models and 9 benchmarks for about $40,000, a public anchor for what runs cost.
  • Expert ground truth needs review: MMLU-Redux found errors in 57% of analyzed virology questions, against an estimated 6.49% across MMLU.
  • Models change under a fixed name: GPT-4's accuracy on a prime-number task fell from 84% to 51% between March and June 2023.
  • In an illustrative 300-task model, upkeep of 290 expert hours a year passes the 320-hour build during year two.

What does a reliable agent workflow cost in data, and which part recurs?

Most of it is evaluation: the tasks, expert answers, repeated runs and upkeep that show an agent completes the workflow correctly every time, not once. Training examples can be bought once. The evidence that an agent stays reliable has to be produced again whenever the model, the business rules or the work itself changes.

For an investor, the decision is which data businesses capture that recurring spend and which sell a one-time archive. The evidence that would answer it has three parts: public data on what agent evaluation consumes in runs and expert time, public evidence on what forces a test set to be rebuilt, and supplier records showing who pays for each line. Published sources cover the first two; for the third, this article offers a transparent model, because no buyer or supplier has published spending data.

Reliable here means the workflow succeeds consistently across repeated attempts and keeps succeeding after changes, a stricter bar than a high score on a single pass.

Why reliability multiplies runs, not just tasks

Because agents are stochastic, one pass says little about the next, so each task is run several times and every configuration change restarts the cycle. Cost grows with tasks multiplied by repetitions multiplied by configurations.

In AI Agents That Matter (2024), Kapoor and colleagues at Princeton note that the language models under most agents are stochastic, so calling them repeatedly can raise accuracy; fair comparisons therefore have to hold cost fixed and report accuracy and cost together. A 2026 follow-up from the same group, Towards a Science of AI Agent Reliability, proposes twelve metrics across four dimensions, consistency, robustness, predictability and safety, and reports that recent capability gains have yielded only small improvements in reliability.

Running those measurements is not free. The Holistic Agent Leaderboard paper reports a validation run of 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science and customer service, at a total cost of about $40,000, and notes that a single benchmark could take weeks to run serially. Those figures cover public benchmarks; a company's private workflow adds its own tasks on top.

The reliability data bill: five lines, three of them recurring

Five lines make up the data cost of a reliable workflow, and only the first two are paid once per task. The table names what each line buys, whether it recurs, the public evidence that it is real, and who is placed to supply it.

Read the recurrence column first. A supplier paid only for task construction competes with every archive and generator. A supplier embedded in repeated trials, maintenance or freshness sells into a budget that renews with each model release and each rule change.

The reliability data bill: five lines, three of them recurring
LineWhat it buysPaid once or recurringPublic evidenceWho can supply it
Task constructionReplayable starting states, allowed actions and acceptance checks drawn from real workOnce per taskSWE-bench-Live gives every task its own Docker imageRecord holders working with environment builders
Expert ground truthExpected outcomes or rubrics written and adjudicated by qualified peopleOnce per task, again on revisionHealthBench: 262 physicians wrote rubrics for 5,000 conversationsDomain experts, ideally people who did the work
Repeated trialsSeveral runs per task per configuration to measure consistencyEvery model, prompt or tool changeHAL: 21,730 rollouts for about $40,000Compute, usually paid by the buyer
Regression maintenanceRe-review and re-runs when models, rules or tools changeContinuousGPT-4 accuracy on a prime-number task fell from 84% to 51% in three months of 2023Record holders, who see rule changes first
Freshness and failure intakeNew cases to replace contaminated ones and capture production failuresContinuousSWE-bench-Live: 1,319 tasks from GitHub issues created since 2024Companies still doing the work

Where expert review earns its cost

Expert review is the line that turns a plausible answer into ground truth, and skipping it leaves errors that distort every comparison built on the set. Two public benchmarks show the scale of the work and the price of skipping it.

OpenAI built HealthBench with 262 physicians who have practiced in 60 countries: 5,000 health conversations, each graded against a rubric written by physicians for that example, for 48,562 unique criteria in all. A model does the grading. The expensive human work went into deciding what a good answer contains, a split between expert authoring and model grading that business workflows are likely to copy.

The authors of Are We Done with MMLU? show what happens without that review. They report that 57% of the questions they analyzed in MMLU's virology subset contain errors, estimate that 6.49% of MMLU questions overall are flawed, and re-annotated 5,700 questions across all 57 subjects to build MMLU-Redux, finding significant discrepancies with previously reported model performance. In a business workflow, an unreviewed expected answer is a policy interpretation nobody checked.

Review hours scale with the number of tasks and again with every revision. The reviewers who reach a correct answer fastest are often the people who did the original work, which favors suppliers close to the record holder over those who only hold copies of records.

What forces a reliable workflow to be tested again

Four events force a re-test: a model change, a rule change, a tool or interface change, and contamination of the test set. Each arrives on someone else's schedule, which is why maintenance is a standing cost rather than a project.

Model changes can move results even under an unchanged model name. Chen, Zaharia and Zou compared the March and June 2023 versions of GPT-4 and GPT-3.5 and report that GPT-4's accuracy at telling prime from composite numbers fell from 84% to 51%, while other behaviors shifted in both directions. A workflow that passed in March needed a fresh run to show it still passed in June.

Contamination is the second clock. The SWE-bench-Live authors argue that static benchmarks risk leaking into pretraining data, and they answer with an automated pipeline that builds tasks from GitHub issues created since 2024, starting with 1,319 tasks across 93 repositories, each in its own Docker image. Automated curation is what makes regular refreshes affordable for code. Business workflows have no comparable public source of fresh tasks, so replacement cases have to come from companies still doing the work.

Rule and tool changes start inside the business: a new approval limit, a revised refund rule, a redesigned ticket form. Each one invalidates some expected answers, and finding which ones takes the rule's version history and a list of the cases it touched.

An illustrative three-year model in expert hours and runs

In this illustrative model the build takes 320 expert hours and upkeep takes 290 hours a year, so cumulative upkeep passes the build during year two and accounts for 73% of expert hours over three years. Every input is an assumption chosen so the arithmetic can be checked. Change the inputs and the crossover moves; the structure, a one-time build set against an upkeep bill that renews, does not.

Inputs, all illustrative: 300 tasks; 45 minutes of expert authoring and 15 minutes of second-expert review per task; 20 minutes of adjudication on the 20% of tasks where reviewers disagree; 5 runs per task for each of 6 configurations per release, with 4 releases a year; rule changes touching 10% of tasks each quarter at 45 minutes per revision, review included; 25% of tasks replaced each year at the full authoring, review and adjudication rates; and 10 production failures a month turned into new tasks at 60 minutes each.

Outputs: the build is 225 hours of authoring, 75 of review and 20 of adjudication. Yearly upkeep is 90 hours of rule revisions, 80 of refresh and 120 of failure intake. Runs come to 9,000 per release and 36,000 a year, a compute line the buyer usually pays, which scales directly with the number of repetitions chosen. The figure plots cumulative build and upkeep hours over three years.

An illustrative three-year model in expert hours and runs
LineIllustrative inputLaunch (expert hours)Each later year (expert hours)
Authoring300 tasks at 45 minutes each225Inside refresh and failure intake
Review and adjudication15 minutes per task, plus 20 minutes on 20% of tasks95Inside refresh and rule revisions
Rule revisions10% of tasks each quarter at 45 minutes090
Refresh against contamination25% of tasks a year at full build rates080
Failure intake10 failures a month at 60 minutes each0120
Total expert hoursRuns: 9,000 per release, 36,000 a year320290
A line chart of cumulative expert hours over three years: the build holds flat at 320 hours while upkeep rises by 290 hours a year, crossing the build during year two and reaching 870 hours.
Illustrative: build versus cumulative upkeep in expert hours for a 300-task evaluation set, using the inputs in the table.

Does an archive of records lower the bill or just move it?

It lowers the bill only when the records already carry the state and the outcome a test needs; otherwise the cost moves into reconstruction. This is the strongest alternative explanation for why real records might not earn a premium: a record collection may lack the environment or outcome evidence an agent needs.

Map each candidate record against the task's requirements: the starting state the agent sees, the actions it may take, the constraints that bind those actions, and an outcome a grader can verify. Records that hold all four cut the construction and ground-truth lines. Records with decisions but no state snapshot or outcome force a builder to rebuild context, and a buyer will net that cost against the record's price.

Records earn a recurring role in maintenance and freshness. A company that keeps operating produces new cases, rule changes and failures every quarter, which is the supply a regression program consumes. A static archive, however large, serves the build once and then ages as rules and models move.

What could make the recurring-cost thesis wrong

Four developments would shrink the recurring lines or pull them in-house, and each has an early public signal to watch.

  • Model graders absorb review. HealthBench already grades with a model; if rubric writing can also be automated without losing accuracy, expert hours fall and the ground-truth line drifts toward the cost of compute.
  • Curation pipelines automate freshness. SWE-bench-Live builds and validates tasks from public issues automatically; a similar pipeline running over a company's own systems would leave outside suppliers out of the freshness line.
  • Model behavior stabilizes. If providers pin versions and change them rarely, model-driven re-runs become infrequent, although rule and tool changes would continue to force some.
  • Buyers keep the work internal. Enterprises that own both the workflow and its records may build and maintain their own tests, leaving outside suppliers only task construction, the least defensible line.

Diligence questions on recurring evaluation data, put to SourceX as well

Ask which lines of the bill a supplier is paid for, how often, and on what evidence the payments renew; then put the same questions to SourceX.

SourceX offers one way to test this thesis, and nothing here treats it as proof. The company arranges licenses for business records that their owners decide to offer AI developers, and it runs each deal from qualifying the business through rights review, contracting, preparation, delivery and payment; nothing moves until the owner has approved the scope and terms and signed. Because buyer datasets are sourced from U.S. businesses on request rather than kept as inventory, recurring refreshes would depend on suppliers that keep producing cases, a point to examine deal by deal.

SourceX has published no figures on renewals, deliveries or revenue, and this article estimates none. Its commercial metrics are discussed with investors who request materials, and its Rule 506(c) raise accepts only accredited investors whose status is verified. Questions worth asking:

  • Which of the five lines does the supplier sell, and which recur under contract rather than by expectation?
  • How are records turned into replayable starting states, and what share of sampled records needed reconstruction?
  • Who writes and adjudicates expected answers, and how is agreement between reviewers measured?
  • When a rule changes at the record holder, how fast can the affected tasks be found and revised, and who pays for it?
  • How many runs per task does the buyer require before calling a workflow reliable, and how is run-to-run variance reported?
  • What is the contamination plan: refresh rate, holdout policy and the provenance of new cases?

Methods note: model inputs, sources and limits

The five-line bill and the three-year model were built for this article on 2026-10-27. Public figures come from the Holistic Agent Leaderboard paper, AI Agents That Matter, Towards a Science of AI Agent Reliability, OpenAI's HealthBench announcement, Are We Done with MMLU?, Chen, Zaharia and Zou (2023) and SWE-bench Goes Live!, each verified through search excerpts on arxiv.org or openai.com and dated in the text.

Every input in the hours model is illustrative and chosen so the arithmetic can be checked; none is a measurement of any company, buyer or SourceX deal. Not measured: real authoring and review times for business workflows, how often enterprise rules change, buyer budgets for evaluation, the split between in-house and outside spending, and compute prices. The model counts expert hours and runs, not money, so it says nothing about margins.

Frequently asked questions

How many runs per task are needed to call an agent workflow reliable?

No standard number exists. Because agents are stochastic, reliability has to be estimated from repeated attempts on the same task, and the number of attempts should be reported with the result. Princeton researchers recommend reporting accuracy together with cost and measuring consistency directly rather than quoting one best run.

Why do agent evaluation sets need ongoing maintenance?

Models change, sometimes under the same name, business rules change, and public test items leak into training data. Chen, Zaharia and Zou found GPT-4's accuracy on one task fell from 84% to 51% between March and June 2023, and SWE-bench-Live builds tasks from recent GitHub issues to stay ahead of contamination.

Can model graders replace expert reviewers in agent evaluation?

Partly. HealthBench grades responses with a model, but 262 physicians wrote the 48,562 rubric criteria it grades against. Models can apply criteria at scale; deciding what the right answer is, and settling disputes about it, still needs people who know the work.

What share of agent data spending goes to evaluation rather than training?

No public data measures it. This article's three-year model is illustrative: under its assumptions, recurring upkeep is 73% of expert hours, but the share depends on how often rules and models change and how many runs a buyer requires.

Does SourceX publish evaluation cost or renewal figures?

No. SourceX has not published cost, renewal, delivery or revenue figures, and this article contains none. Investors can request materials through sourcex.si/investors and ask how recurring supply would work; any offering is made only to verified accredited investors.

Sources

  • The Holistic Agent Leaderboard paper reports 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science and customer service at a total cost of about $40,000, and says a single benchmark could take weeks to run serially. Source
  • AI Agents That Matter (Kapoor et al., 2024) argues that because agents' underlying models are stochastic, repeated calls can raise accuracy, so evaluations should control cost and optimize accuracy and cost jointly. Source
  • Towards a Science of AI Agent Reliability proposes twelve metrics across consistency, robustness, predictability and safety and finds recent capability gains have yielded only small improvements in reliability. Source
  • OpenAI's HealthBench was built with 262 physicians who have practiced in 60 countries: 5,000 conversations with physician-written rubrics totaling 48,562 unique criteria, graded by a model. Source
  • Are We Done with MMLU? reports errors in 57% of analyzed virology questions, estimates 6.49% of MMLU questions contain errors, and re-annotated 5,700 questions across 57 subjects as MMLU-Redux, finding significant discrepancies with reported model performance. Source
  • Chen, Zaharia and Zou report that GPT-4's accuracy at identifying prime versus composite numbers fell from 84% (March 2023) to 51% (June 2023), with other behaviors drifting. Source
  • SWE-bench Goes Live! presents SWE-bench-Live, 1,319 tasks from GitHub issues created since 2024 across 93 repositories, each with its own Docker image, built by an automated curation pipeline to resist contamination. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies