Skip to content

AI uses for records

Enterprise agent cost-quality benchmark: model, data or oversight?

By SourceX Editorial · Updated

10 sections

Short answer

An enterprise agent cost-quality benchmark should vary one lever at a time, model, data or human oversight, across matched configurations on a fixed set of business tasks. Count every cost from tokens to review minutes, score success, error severity, latency and review effort separately, and plot the nondominated setups. Then label each change: it shifts the frontier, trades one cost for another, or sits in the noise. No results are published yet.

Agent run

Enterprise agent cost

This page covers enterprise agent cost-quality benchmark: model, data or oversight.Illustrative

Key takeaways

  • Benchmark configurations, not models: the model, the business data it sees, the inference mode and the review policy each change cost and quality.
  • Inference-only cost comparisons leave out oversight, iteration and integration; OpenAI's GDPval release says so about its own 100x figures.
  • Eight matched configurations crossing model, data and oversight show each lever's effect and whether levers interact.
  • Keep success, severity, latency and review effort as separate outcomes, and test oversight with seeded errors instead of assuming it works.
  • Label every change a shift, a trade or noise, report frontier membership as a resampled probability, and state the volume at which each verdict flips.

What should an enterprise agent cost-quality benchmark compare?

It should compare configurations, not models. A configuration is the whole setup that does the work: the model and its settings, the business data it can see, the inference mode and the human oversight wrapped around it. The benchmark holds one set of business tasks fixed, changes one lever at a time and records what each change bought and what it cost.

That is a different question from choosing between two finished agents. The cost per verified success worksheet on this site ranks agents that already exist by the full cost of a finished task. This design asks which lever an AI team should pull next: a larger model, better context drawn from real business records, or a different review policy. A team that only swaps models never learns that a smaller model with good records, or a narrower review rule, would have beaten the upgrade.

Everything below is a research protocol. SourceX has not run this study and has published no results from it; every value in the tables and the figure is a definition, a design parameter or labeled Illustrative.

Why the model bill understates what a business task costs

The model bill counts inference, while a business pays for the whole configuration: preparing context, checking output, fixing what slipped through and waiting for the result. A benchmark that prices tokens alone rewards configurations that quietly move work onto people.

OpenAI's GDPval release (2025), which grades model deliverables on tasks from 44 occupations in the nine industries contributing most to U.S. GDP, states the gap plainly. It reports that frontier models completed GDPval tasks roughly 100x faster and 100x cheaper than industry experts, then cautions that those figures reflect pure model inference time and API billing rates and do not capture the human oversight, iteration and integration steps real workplaces need. That caveat is this benchmark's agenda: measure the steps the headline leaves out.

Agent researchers make the same point about accuracy. The README of the Holistic Agent Leaderboard (HAL) harness from Princeton argues that benchmark evaluations tend to focus on accuracy while ignoring costs, and asks what a 1 percent accuracy gain means if the agent is 10 times more expensive. HAL's response was cost-controlled evaluation by default. A business benchmark needs one more column than HAL tracks: the cost of the people in the loop.

Those people are not a perfect safety net. NIST's Generative AI Profile (NIST AI 600-1, 2024) treats human-AI configuration as a risk in its own right, including automation bias and over-reliance. A review step that approves nearly everything adds cost without adding quality, so the benchmark measures what reviewers actually catch instead of assuming it.

The configuration table: eight matched setups, one lever at a time

Cross three levers at two levels each and run all eight combinations on the same tasks. A full two-by-two-by-two design estimates each lever's effect and shows whether levers interact, for example whether business context helps the smaller model more than the larger one.

The levels below are an example to adapt. Whatever levels a team chooses, the preregistration records them precisely enough that a second team could rebuild every configuration from the table alone: exact model identifiers and settings, the record sets and how they were prepared, and the review rule with its time cap.

Freeze the tasks first and keep them away from the records the agent can see. Draw tasks from completed work with recorded end states, weight task families by their share of production volume, and give the set a version number and date before any configuration touches it.

Licensed records with outcomes means past cases from the same workflow, such as resolved tickets, settled claims or reconciled invoices, supplied as retrieval context or worked examples. Any record that shares a customer, case or document with an evaluation task is removed before the run; otherwise the data condition leaks answers and its gain is an artifact.

Record a human reference point on the same tasks. The human baseline data guide on this site covers how to capture how long trained staff take and how often they get the end state right, which turns the frontier into a comparison with current practice, not only among machines.

  • Task manifest: task family, source system, required end state and the evidence that confirms it.
  • Exclusion log: every record removed from the data condition because it overlapped an evaluation task, with the matching rule used.
  • Data condition card: record types, date range, volume, preparation steps and the license scope that permits evaluation use.
  • Human reference: completion time and verified success for a sample of tasks done by staff under normal conditions.
The configuration table: eight matched setups, one lever at a time
ConfigModel tierData conditionOversight policyWhat it isolates
C1Smaller modelTask instructions onlySpot-check 1 in 10 outputsBaseline: the cheapest setup that could plausibly work
C2Larger modelTask instructions onlySpot-check 1 in 10 outputsModel effect with no business context
C3Smaller modelInstructions plus licensed records with outcomesSpot-check 1 in 10 outputsData effect on the cheaper model
C4Larger modelInstructions plus licensed records with outcomesSpot-check 1 in 10 outputsWhether data and model gains add up or overlap
C5Smaller modelTask instructions onlyReview every high-severity actionOversight effect without context
C6Larger modelTask instructions onlyReview every high-severity actionWhat oversight costs when the model is stronger
C7Smaller modelInstructions plus licensed records with outcomesReview every high-severity actionCheap model, rich context, gated actions
C8Larger modelInstructions plus licensed records with outcomesReview every high-severity actionThe most expensive setup and the ceiling to beat

Which costs go in the ledger, and how are fixed costs spread?

Every cost that differs between configurations goes in, measured per task where possible and spread over a stated task volume where not. The ledger has to make a one-time data cost and a per-task review cost comparable, so the allocation rule is written down before the run.

Log usage with standard fields so anyone can rebuild the ledger. The OpenTelemetry generative AI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens for input served from a provider-managed cache; the attributes are still marked as in development, so store the convention version with the data.

Inference mode is a cost lever in its own right, which is why every result carries a pricing date. Anthropic's batch processing documentation, read on October 26, 2026, says batch requests cost 50 percent less, with most batches finishing in less than an hour and batches expiring if processing does not complete within 24 hours. The same model and prompts can cost half as much in a configuration that can wait.

The data line is where cost-quality studies most often cheat by omission. A configuration that uses licensed records looks free per task if the license and preparation are left out, and hopeless if they are charged to the first hundred tasks. Show every result at the expected volume and at half and double it, so readers can see whether the data condition stays on the frontier at their own scale.

Which costs go in the ledger, and how are fixed costs spread?
Cost lineUnit measuredHow it is capturedPer task or fixedAllocation rule
Model inferenceInput, output and cached tokensUsage logged per runPer taskLogged usage times contracted rates on a stated pricing date
Tools and external callsCalls by typeTool-call logsPer taskCount times unit cost, including any verifier model calls
Data acquisition and preparationLicense cost and preparation hoursContracts and time recordsFixedSpread over expected task volume, with half and double volume shown
Context build and refreshIndexing runs and refresh cyclesPipeline logsFixed per refreshSpread over the tasks served between refreshes
Human reviewReviewer minutesReview tool timestampsPer taskMinutes times loaded hourly cost, with edits counted apart from approvals
Failure remediationRework minutes and reversalsRework and reversal recordsPer failed taskMeasured or estimated per task family, never one average
Waiting timeMinutes from request to verified completionRun and review timestampsPer taskPriced only where the business assigns a cost to delay

Score success, severity, latency and review effort separately

Report each outcome on its own for every configuration, because each one fails a business in a different way. Collapsing them into a single score hides exactly the trades this benchmark exists to expose.

Verification looks at the world, not the transcript. Anthropic's guide to agent evals (January 2026) separates the transcript, the full record of a trial, from the outcome, the final state the trial leaves in the environment: a booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database. Apply the same rule to business work; a refund counts when the ledger shows it, not when the agent says so.

Seed known errors into a sample of outputs before they reach reviewers, who stay blind to the configuration. A review policy that catches few seeded errors is a cost line rather than a control, and configurations that lean on it are scored as if it were absent. Severity tiers and latency timing reuse the definitions in this site's guides on severity weights and on latency and rework, so results stay comparable across studies.

Score success, severity, latency and review effort separately
OutcomeDefinitionDenominatorWhy it stays separate
Verified successRequired end state confirmed in the system of record, not in the agent's own reportAll tasks attemptedThe quality axis of the frontier
Severity-weighted errorsEach failure scored by business consequence tier, with critical errors also countedAll tasks attemptedTwo setups with equal success can fail on very different tasks
LatencyMedian and 90th percentile minutes from request to verified completion, review includedCompleted tasks, with open cases reportedReview policies can move latency more than model choice does
Review effortReviewer minutes per task and the share of outputs edited or rejectedTasks routed to reviewShows where quality was bought with people
Reviewer catch rateShare of seeded known errors that reviewers stoppedSeeded error itemsTests whether oversight works or only adds cost

Run rules: budgets, retries and how much a reviewer may help

Write the limits down before the first run and apply them identically to all eight configurations. Without fixed budgets, the expensive configuration wins by spending more, and the comparison measures generosity rather than design.

Repeated trials matter because averages hide inconsistency. The original tau-bench repository's airline leaderboard lists one tool-calling setup at 0.460 when it needs to succeed once (pass^1) and 0.225 when it must succeed on all four of four trials (pass^4). Anthropic's evals guide describes the same pattern: pass^k falls as k rises because consistency across more trials is harder. A workflow that runs every day behaves more like the second number, so report both.

  • Token and tool budget: a hard cap per task on tokens and on tool calls, with the share of tasks that hit each cap reported.
  • Retry policy: the number of attempts production would allow; every attempt is costed and a task counts as one success at most.
  • Trials: at least three independent trials per task and configuration, so reliability is reported alongside average success.
  • Reviewer authority: approve, reject or edit, with edits logged and their minutes charged to the configuration; a task a reviewer finishes by editing counts as human work, not agent success.
  • Review time cap: a stated maximum per item, after which the item escalates and is scored unresolved.
  • Blinding: reviewers and graders never see which configuration produced an output.
  • Freeze and version: task set, record sets, prompts, model identifiers and prices recorded with dates; any change starts a new run.

Drawing the frontier when configurations are close

A configuration is on the frontier when no other configuration delivers higher verified success at equal or lower full cost per task while also meeting the latency and critical-error limits set in advance. Treat latency and critical errors as constraints rather than extra axes: any configuration that breaks a limit is dropped before the frontier is drawn, which keeps the chart readable.

Compare configurations on the same tasks and analyze per-task differences. Anthropic's note on statistical evals (November 2024) recommends analyzing paired differences and clustering standard errors when questions are not independent. For business tasks the natural cluster is the task family or the source case, because several tasks drawn from one customer history share hidden context.

Then report frontier membership as a probability, not a verdict. Resample task clusters a fixed number of times set in the preregistration, redraw the frontier each time and record the share of resamples in which each configuration is nondominated. A configuration that is on the frontier in nearly every resample is a finding; one that is on it about half the time is a tie, and the write-up says so.

The figure shows the form of the output with Illustrative positions: eight configurations, the frontier through the nondominated ones, and the two kinds of movement the next section classifies.

A cost-against-success chart of eight agent configurations with a frontier line through the nondominated ones, one arrow marking a shift that raises success at similar cost and one marking a trade along the frontier.
Frontier, shift and trade for eight configurations. Illustrative: positions are examples, not measured results.

Shift, trade or noise: what did each change actually buy?

Give every lever change one of three verdicts. A shift moves the frontier: the new configuration dominates the old, with more success at no more cost or equal success for less. A trade moves along it, with quality and cost rising or falling together. Noise means the change sits inside the uncertainty from the resampling step.

Two cautions keep the labels honest. A retrieval gain is not a task gain: Anthropic's contextual retrieval figures measure how often the right passage was found, so the benchmark must confirm that verified success rose too. And a shift at one volume can be a trade at another, because data and context are fixed costs; report the task volume at which each verdict flips.

Interactions are findings in their own right. If business context lifts the smaller model more than the larger one, the cheapest frontier configuration may be the smaller model with good records, which a model-only comparison would never reveal.

Shift, trade or noise: what did each change actually buy?
ChangeEffect to testVerdict whenPublic evidence, dated
Batch instead of real-time inferenceSame outputs at lower cost, slower resultsShift if the task tolerates the wait; trade if latency is a limitAnthropic batch docs, read October 2026: 50 percent lower cost, most batches under an hour
Context built from licensed recordsHigher success, added fixed costShift if success rises at equal full cost at the expected volumeAnthropic, September 2024: contextual retrieval cut failed retrievals by 49 percent, 67 percent with reranking
Larger modelHigher success, higher per-task costUsually a trade; a shift only if fewer failures cut remediation enoughOpenAI GDPval, 2025: its speed and cost comparison excludes oversight and iteration
Review every high-severity actionFewer critical errors, more minutes and latencyTrade, unless seeded errors show reviewers miss most issuesNIST AI 600-1, 2024: automation bias and over-reliance are named risks
More retries per taskHigher pass rate, more cost per successTrade; check consistency across trialstau-bench repository: pass^1 of 0.460 against pass^4 of 0.225 for one setup

What this design cannot show, and what has not been measured

It describes one task set, one price list and one review team at one point in time. It does not rank models or vendors in general, and its verdicts expire when prices, model versions or the workload mix change. SourceX has not conducted this study, and no results, rankings or effect sizes from it exist yet.

At a glance

What this design cannot show, and what has not been measured

Publish negative results as readily as positive ones. A data condition that fails to move the frontier is useful evidence for the next team deciding where to spend, and a study that reports only shifts invites the suspicion that trades were left out.

  • Not measured: long-run effects such as reviewer fatigue, workload drift or the cost of maintaining context for a year.
  • Not covered: tasks with no verifiable end state, where success rests on judgment no record can confirm.
  • Not transferable: the effect of licensed records in one workflow says little about another; each data condition is tested on its own tasks.
  • Not a price benchmark: costs use the team's own dated rates, so cross-company comparisons need a shared price basis.

Sourcing the data condition: past cases with outcomes you can verify

The data condition needs records from the same kind of work the agent will do, with outcomes recorded in a system of record and enough history to keep evaluation tasks apart from context. As a general guide from SourceX, buyers want years of consistent history, usually two or more, held in one system with tens of thousands of records or more, while smaller sets can qualify when the work is rare; every case is assessed on its own.

There is no SourceX shelf of past cases to pick from: datasets are sourced from U.S. businesses when a buyer asks. Through the request form, a team running this benchmark can set out its task families, the outcome evidence it needs and the acceptance test it will apply, alongside the usual scope, volume, language, format, timing and licensing needs. Whether matching records exist, and whether they move your frontier, is exactly what the benchmark would test.

Frequently asked questions

What is an enterprise agent cost-quality benchmark?

It is a controlled comparison of complete agent configurations, meaning model, business data, inference mode and human oversight, on one fixed set of business tasks. Each configuration is scored on verified success, error severity, latency and review effort against its full cost per task. The configurations that no other setup beats on both cost and success form the frontier.

How is a cost-quality frontier different from cost per verified success?

Cost per verified success ranks finished agents by a single ratio. A cost-quality frontier keeps cost and quality as separate axes, applies latency and critical-error limits as constraints, and shows which changes move the frontier and which only trade one cost for another. The two work together: the ratio picks an agent, the frontier tells you what to change next.

Should human review time count as part of an agent's cost?

Yes. Review minutes, edits and rework are costs the configuration creates, and leaving them out favors setups that move work onto people. OpenAI's GDPval release makes the same caveat about its own speed and cost comparison, which reflects inference time and API billing only.

How many trials per task does a fair comparison need?

Enough to report reliability as well as averages; three or more independent trials per task and configuration is a reasonable floor. Repeated trials expose inconsistency that one run hides, as the falling pass^k results in the tau-bench repository show. Fix the number in the preregistration so it cannot change after results arrive.

Has SourceX run the cost-quality benchmark or published any scores?

No. This page is a study design. SourceX has not run the benchmark or published any results, and every value on the page is a definition, a design parameter or labeled Illustrative.

Sources

  • OpenAI's GDPval covers tasks from 44 occupations in the top nine industries contributing to U.S. GDP and reports frontier models completing its tasks roughly 100x faster and 100x cheaper than experts, figures that reflect inference time and API billing only and exclude human oversight, iteration and integration. Source
  • The Holistic Agent Leaderboard harness README says benchmark evaluations tend to focus on accuracy while ignoring costs, asks what 1% higher accuracy means at 10x the cost, and describes cost-controlled evaluation by default. Source
  • NIST's Generative AI Profile (NIST AI 600-1) defines human-AI configuration as a risk that includes automation bias and over-reliance. Source
  • Anthropic's guide to agent evals (January 2026) defines the outcome as the final environment state, not the transcript, using a booking agent whose success depends on a reservation existing in the database, and explains that pass^k falls as k rises. Source
  • Anthropic's note on statistical model evals (November 2024) recommends clustering standard errors for non-independent questions and analyzing paired differences between models. Source
  • Anthropic reported in September 2024 that contextual retrieval reduced failed retrievals by 49%, and by 67% when combined with reranking, on its tests. Source
  • Anthropic's batch processing documentation says the Message Batches API reduces costs by 50%, most batches finish in less than an hour, and batches expire if processing does not complete within 24 hours. Source
  • The original tau-bench repository's airline leaderboard lists a tool-calling Claude 3.5 Sonnet (October 2024) setup at 0.460 pass^1 and 0.225 pass^4. Source
  • The OpenTelemetry GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens, marked as development status. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies