AI uses for records
Enterprise agent cost-quality benchmark: model, data or oversight?
By SourceX Editorial · Updated
10 sections
Short answer
An enterprise agent cost-quality benchmark should vary one lever at a time, model, data or human oversight, across matched configurations on a fixed set of business tasks. Count every cost from tokens to review minutes, score success, error severity, latency and review effort separately, and plot the nondominated setups. Then label each change: it shifts the frontier, trades one cost for another, or sits in the noise. No results are published yet.
Agent run
Enterprise agent cost
Key takeaways
- Benchmark configurations, not models: the model, the business data it sees, the inference mode and the review policy each change cost and quality.
- Inference-only cost comparisons leave out oversight, iteration and integration; OpenAI's GDPval release says so about its own 100x figures.
- Eight matched configurations crossing model, data and oversight show each lever's effect and whether levers interact.
- Keep success, severity, latency and review effort as separate outcomes, and test oversight with seeded errors instead of assuming it works.
- Label every change a shift, a trade or noise, report frontier membership as a resampled probability, and state the volume at which each verdict flips.
What should an enterprise agent cost-quality benchmark compare?
It should compare configurations, not models. A configuration is the whole setup that does the work: the model and its settings, the business data it can see, the inference mode and the human oversight wrapped around it. The benchmark holds one set of business tasks fixed, changes one lever at a time and records what each change bought and what it cost.
That is a different question from choosing between two finished agents. The cost per verified success worksheet on this site ranks agents that already exist by the full cost of a finished task. This design asks which lever an AI team should pull next: a larger model, better context drawn from real business records, or a different review policy. A team that only swaps models never learns that a smaller model with good records, or a narrower review rule, would have beaten the upgrade.
Everything below is a research protocol. SourceX has not run this study and has published no results from it; every value in the tables and the figure is a definition, a design parameter or labeled Illustrative.
Why the model bill understates what a business task costs
The model bill counts inference, while a business pays for the whole configuration: preparing context, checking output, fixing what slipped through and waiting for the result. A benchmark that prices tokens alone rewards configurations that quietly move work onto people.
OpenAI's GDPval release (2025), which grades model deliverables on tasks from 44 occupations in the nine industries contributing most to U.S. GDP, states the gap plainly. It reports that frontier models completed GDPval tasks roughly 100x faster and 100x cheaper than industry experts, then cautions that those figures reflect pure model inference time and API billing rates and do not capture the human oversight, iteration and integration steps real workplaces need. That caveat is this benchmark's agenda: measure the steps the headline leaves out.
Agent researchers make the same point about accuracy. The README of the Holistic Agent Leaderboard (HAL) harness from Princeton argues that benchmark evaluations tend to focus on accuracy while ignoring costs, and asks what a 1 percent accuracy gain means if the agent is 10 times more expensive. HAL's response was cost-controlled evaluation by default. A business benchmark needs one more column than HAL tracks: the cost of the people in the loop.
Those people are not a perfect safety net. NIST's Generative AI Profile (NIST AI 600-1, 2024) treats human-AI configuration as a risk in its own right, including automation bias and over-reliance. A review step that approves nearly everything adds cost without adding quality, so the benchmark measures what reviewers actually catch instead of assuming it.
The configuration table: eight matched setups, one lever at a time
Cross three levers at two levels each and run all eight combinations on the same tasks. A full two-by-two-by-two design estimates each lever's effect and shows whether levers interact, for example whether business context helps the smaller model more than the larger one.
The levels below are an example to adapt. Whatever levels a team chooses, the preregistration records them precisely enough that a second team could rebuild every configuration from the table alone: exact model identifiers and settings, the record sets and how they were prepared, and the review rule with its time cap.
Freeze the tasks first and keep them away from the records the agent can see. Draw tasks from completed work with recorded end states, weight task families by their share of production volume, and give the set a version number and date before any configuration touches it.
Licensed records with outcomes means past cases from the same workflow, such as resolved tickets, settled claims or reconciled invoices, supplied as retrieval context or worked examples. Any record that shares a customer, case or document with an evaluation task is removed before the run; otherwise the data condition leaks answers and its gain is an artifact.
Record a human reference point on the same tasks. The human baseline data guide on this site covers how to capture how long trained staff take and how often they get the end state right, which turns the frontier into a comparison with current practice, not only among machines.
- Task manifest: task family, source system, required end state and the evidence that confirms it.
- Exclusion log: every record removed from the data condition because it overlapped an evaluation task, with the matching rule used.
- Data condition card: record types, date range, volume, preparation steps and the license scope that permits evaluation use.
- Human reference: completion time and verified success for a sample of tasks done by staff under normal conditions.
| Config | Model tier | Data condition | Oversight policy | What it isolates |
|---|---|---|---|---|
| C1 | Smaller model | Task instructions only | Spot-check 1 in 10 outputs | Baseline: the cheapest setup that could plausibly work |
| C2 | Larger model | Task instructions only | Spot-check 1 in 10 outputs | Model effect with no business context |
| C3 | Smaller model | Instructions plus licensed records with outcomes | Spot-check 1 in 10 outputs | Data effect on the cheaper model |
| C4 | Larger model | Instructions plus licensed records with outcomes | Spot-check 1 in 10 outputs | Whether data and model gains add up or overlap |
| C5 | Smaller model | Task instructions only | Review every high-severity action | Oversight effect without context |
| C6 | Larger model | Task instructions only | Review every high-severity action | What oversight costs when the model is stronger |
| C7 | Smaller model | Instructions plus licensed records with outcomes | Review every high-severity action | Cheap model, rich context, gated actions |
| C8 | Larger model | Instructions plus licensed records with outcomes | Review every high-severity action | The most expensive setup and the ceiling to beat |
Which costs go in the ledger, and how are fixed costs spread?
Every cost that differs between configurations goes in, measured per task where possible and spread over a stated task volume where not. The ledger has to make a one-time data cost and a per-task review cost comparable, so the allocation rule is written down before the run.
Log usage with standard fields so anyone can rebuild the ledger. The OpenTelemetry generative AI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens for input served from a provider-managed cache; the attributes are still marked as in development, so store the convention version with the data.
Inference mode is a cost lever in its own right, which is why every result carries a pricing date. Anthropic's batch processing documentation, read on October 26, 2026, says batch requests cost 50 percent less, with most batches finishing in less than an hour and batches expiring if processing does not complete within 24 hours. The same model and prompts can cost half as much in a configuration that can wait.
The data line is where cost-quality studies most often cheat by omission. A configuration that uses licensed records looks free per task if the license and preparation are left out, and hopeless if they are charged to the first hundred tasks. Show every result at the expected volume and at half and double it, so readers can see whether the data condition stays on the frontier at their own scale.
| Cost line | Unit measured | How it is captured | Per task or fixed | Allocation rule |
|---|---|---|---|---|
| Model inference | Input, output and cached tokens | Usage logged per run | Per task | Logged usage times contracted rates on a stated pricing date |
| Tools and external calls | Calls by type | Tool-call logs | Per task | Count times unit cost, including any verifier model calls |
| Data acquisition and preparation | License cost and preparation hours | Contracts and time records | Fixed | Spread over expected task volume, with half and double volume shown |
| Context build and refresh | Indexing runs and refresh cycles | Pipeline logs | Fixed per refresh | Spread over the tasks served between refreshes |
| Human review | Reviewer minutes | Review tool timestamps | Per task | Minutes times loaded hourly cost, with edits counted apart from approvals |
| Failure remediation | Rework minutes and reversals | Rework and reversal records | Per failed task | Measured or estimated per task family, never one average |
| Waiting time | Minutes from request to verified completion | Run and review timestamps | Per task | Priced only where the business assigns a cost to delay |
Score success, severity, latency and review effort separately
Report each outcome on its own for every configuration, because each one fails a business in a different way. Collapsing them into a single score hides exactly the trades this benchmark exists to expose.
Verification looks at the world, not the transcript. Anthropic's guide to agent evals (January 2026) separates the transcript, the full record of a trial, from the outcome, the final state the trial leaves in the environment: a booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database. Apply the same rule to business work; a refund counts when the ledger shows it, not when the agent says so.
Seed known errors into a sample of outputs before they reach reviewers, who stay blind to the configuration. A review policy that catches few seeded errors is a cost line rather than a control, and configurations that lean on it are scored as if it were absent. Severity tiers and latency timing reuse the definitions in this site's guides on severity weights and on latency and rework, so results stay comparable across studies.
| Outcome | Definition | Denominator | Why it stays separate |
|---|---|---|---|
| Verified success | Required end state confirmed in the system of record, not in the agent's own report | All tasks attempted | The quality axis of the frontier |
| Severity-weighted errors | Each failure scored by business consequence tier, with critical errors also counted | All tasks attempted | Two setups with equal success can fail on very different tasks |
| Latency | Median and 90th percentile minutes from request to verified completion, review included | Completed tasks, with open cases reported | Review policies can move latency more than model choice does |
| Review effort | Reviewer minutes per task and the share of outputs edited or rejected | Tasks routed to review | Shows where quality was bought with people |
| Reviewer catch rate | Share of seeded known errors that reviewers stopped | Seeded error items | Tests whether oversight works or only adds cost |
Run rules: budgets, retries and how much a reviewer may help
Write the limits down before the first run and apply them identically to all eight configurations. Without fixed budgets, the expensive configuration wins by spending more, and the comparison measures generosity rather than design.
Repeated trials matter because averages hide inconsistency. The original tau-bench repository's airline leaderboard lists one tool-calling setup at 0.460 when it needs to succeed once (pass^1) and 0.225 when it must succeed on all four of four trials (pass^4). Anthropic's evals guide describes the same pattern: pass^k falls as k rises because consistency across more trials is harder. A workflow that runs every day behaves more like the second number, so report both.
- Token and tool budget: a hard cap per task on tokens and on tool calls, with the share of tasks that hit each cap reported.
- Retry policy: the number of attempts production would allow; every attempt is costed and a task counts as one success at most.
- Trials: at least three independent trials per task and configuration, so reliability is reported alongside average success.
- Reviewer authority: approve, reject or edit, with edits logged and their minutes charged to the configuration; a task a reviewer finishes by editing counts as human work, not agent success.
- Review time cap: a stated maximum per item, after which the item escalates and is scored unresolved.
- Blinding: reviewers and graders never see which configuration produced an output.
- Freeze and version: task set, record sets, prompts, model identifiers and prices recorded with dates; any change starts a new run.
Drawing the frontier when configurations are close
A configuration is on the frontier when no other configuration delivers higher verified success at equal or lower full cost per task while also meeting the latency and critical-error limits set in advance. Treat latency and critical errors as constraints rather than extra axes: any configuration that breaks a limit is dropped before the frontier is drawn, which keeps the chart readable.
Compare configurations on the same tasks and analyze per-task differences. Anthropic's note on statistical evals (November 2024) recommends analyzing paired differences and clustering standard errors when questions are not independent. For business tasks the natural cluster is the task family or the source case, because several tasks drawn from one customer history share hidden context.
Then report frontier membership as a probability, not a verdict. Resample task clusters a fixed number of times set in the preregistration, redraw the frontier each time and record the share of resamples in which each configuration is nondominated. A configuration that is on the frontier in nearly every resample is a finding; one that is on it about half the time is a tie, and the write-up says so.
The figure shows the form of the output with Illustrative positions: eight configurations, the frontier through the nondominated ones, and the two kinds of movement the next section classifies.
Shift, trade or noise: what did each change actually buy?
Give every lever change one of three verdicts. A shift moves the frontier: the new configuration dominates the old, with more success at no more cost or equal success for less. A trade moves along it, with quality and cost rising or falling together. Noise means the change sits inside the uncertainty from the resampling step.
Two cautions keep the labels honest. A retrieval gain is not a task gain: Anthropic's contextual retrieval figures measure how often the right passage was found, so the benchmark must confirm that verified success rose too. And a shift at one volume can be a trade at another, because data and context are fixed costs; report the task volume at which each verdict flips.
Interactions are findings in their own right. If business context lifts the smaller model more than the larger one, the cheapest frontier configuration may be the smaller model with good records, which a model-only comparison would never reveal.
| Change | Effect to test | Verdict when | Public evidence, dated |
|---|---|---|---|
| Batch instead of real-time inference | Same outputs at lower cost, slower results | Shift if the task tolerates the wait; trade if latency is a limit | Anthropic batch docs, read October 2026: 50 percent lower cost, most batches under an hour |
| Context built from licensed records | Higher success, added fixed cost | Shift if success rises at equal full cost at the expected volume | Anthropic, September 2024: contextual retrieval cut failed retrievals by 49 percent, 67 percent with reranking |
| Larger model | Higher success, higher per-task cost | Usually a trade; a shift only if fewer failures cut remediation enough | OpenAI GDPval, 2025: its speed and cost comparison excludes oversight and iteration |
| Review every high-severity action | Fewer critical errors, more minutes and latency | Trade, unless seeded errors show reviewers miss most issues | NIST AI 600-1, 2024: automation bias and over-reliance are named risks |
| More retries per task | Higher pass rate, more cost per success | Trade; check consistency across trials | tau-bench repository: pass^1 of 0.460 against pass^4 of 0.225 for one setup |
What this design cannot show, and what has not been measured
It describes one task set, one price list and one review team at one point in time. It does not rank models or vendors in general, and its verdicts expire when prices, model versions or the workload mix change. SourceX has not conducted this study, and no results, rankings or effect sizes from it exist yet.
At a glance
What this design cannot show, and what has not been measured
Publish negative results as readily as positive ones. A data condition that fails to move the frontier is useful evidence for the next team deciding where to spend, and a study that reports only shifts invites the suspicion that trades were left out.
- Not measured: long-run effects such as reviewer fatigue, workload drift or the cost of maintaining context for a year.
- Not covered: tasks with no verifiable end state, where success rests on judgment no record can confirm.
- Not transferable: the effect of licensed records in one workflow says little about another; each data condition is tested on its own tasks.
- Not a price benchmark: costs use the team's own dated rates, so cross-company comparisons need a shared price basis.
Sourcing the data condition: past cases with outcomes you can verify
The data condition needs records from the same kind of work the agent will do, with outcomes recorded in a system of record and enough history to keep evaluation tasks apart from context. As a general guide from SourceX, buyers want years of consistent history, usually two or more, held in one system with tens of thousands of records or more, while smaller sets can qualify when the work is rare; every case is assessed on its own.
There is no SourceX shelf of past cases to pick from: datasets are sourced from U.S. businesses when a buyer asks. Through the request form, a team running this benchmark can set out its task families, the outcome evidence it needs and the acceptance test it will apply, alongside the usual scope, volume, language, format, timing and licensing needs. Whether matching records exist, and whether they move your frontier, is exactly what the benchmark would test.
Frequently asked questions
What is an enterprise agent cost-quality benchmark?
It is a controlled comparison of complete agent configurations, meaning model, business data, inference mode and human oversight, on one fixed set of business tasks. Each configuration is scored on verified success, error severity, latency and review effort against its full cost per task. The configurations that no other setup beats on both cost and success form the frontier.
How is a cost-quality frontier different from cost per verified success?
Cost per verified success ranks finished agents by a single ratio. A cost-quality frontier keeps cost and quality as separate axes, applies latency and critical-error limits as constraints, and shows which changes move the frontier and which only trade one cost for another. The two work together: the ratio picks an agent, the frontier tells you what to change next.
Should human review time count as part of an agent's cost?
Yes. Review minutes, edits and rework are costs the configuration creates, and leaving them out favors setups that move work onto people. OpenAI's GDPval release makes the same caveat about its own speed and cost comparison, which reflects inference time and API billing only.
How many trials per task does a fair comparison need?
Enough to report reliability as well as averages; three or more independent trials per task and configuration is a reasonable floor. Repeated trials expose inconsistency that one run hides, as the falling pass^k results in the tau-bench repository show. Fix the number in the preregistration so it cannot change after results arrive.
Has SourceX run the cost-quality benchmark or published any scores?
No. This page is a study design. SourceX has not run the benchmark or published any results, and every value on the page is a definition, a design parameter or labeled Illustrative.
Sources
- OpenAI's GDPval covers tasks from 44 occupations in the top nine industries contributing to U.S. GDP and reports frontier models completing its tasks roughly 100x faster and 100x cheaper than experts, figures that reflect inference time and API billing only and exclude human oversight, iteration and integration. Source
- The Holistic Agent Leaderboard harness README says benchmark evaluations tend to focus on accuracy while ignoring costs, asks what 1% higher accuracy means at 10x the cost, and describes cost-controlled evaluation by default. Source
- NIST's Generative AI Profile (NIST AI 600-1) defines human-AI configuration as a risk that includes automation bias and over-reliance. Source
- Anthropic's guide to agent evals (January 2026) defines the outcome as the final environment state, not the transcript, using a booking agent whose success depends on a reservation existing in the database, and explains that pass^k falls as k rises. Source
- Anthropic's note on statistical model evals (November 2024) recommends clustering standard errors for non-independent questions and analyzing paired differences between models. Source
- Anthropic reported in September 2024 that contextual retrieval reduced failed retrievals by 49%, and by 67% when combined with reranking, on its tests. Source
- Anthropic's batch processing documentation says the Message Batches API reduces costs by 50%, most batches finish in less than an hour, and batches expire if processing does not complete within 24 hours. Source
- The original tau-bench repository's airline leaderboard lists a tool-calling Claude 3.5 Sonnet (October 2024) setup at 0.460 pass^1 and 0.225 pass^4. Source
- The OpenTelemetry GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens, marked as development status. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.