AI uses for records
Business task completion benchmark: verify the result, not the claim
By SourceX Editorial · Updated
10 sections
Short answer
A benchmark verifies actual business outcomes by checking, after each attempt, the system of record, the files produced and the messages sent against a business result defined in advance, while recording the agent's completion claim separately. Crossing claim with verified result yields false completion and side-effect rates. This page gives the protocol and task record; it reports no results.
Evaluation split
Business task completion
Key takeaways
- Score the business result found in systems and files, never the agent's own statement that it finished.
- Record the claim separately so the study can measure false completion, the case where people act on work that does not exist.
- Every task needs state assertions, artifact checks, forbidden side effects and acceptable variants written before any model run.
- Audit graders with planted correct and planted wrong end states before trusting any score.
- Report the primary result on held-out companies, with null-agent baselines, repeated trials and company-clustered intervals.
- This is a method only; no results are reported here.
How can a benchmark tell that a business task was actually done?
A benchmark can tell that a business task was done only by inspecting what the task was supposed to change: the records in the system of record, the files that should exist, and the messages that should have reached a customer or colleague. The check runs after the attempt ends, against a definition of the business result written before any model is tested, and it ignores what the agent says about its own work.
This page is a protocol for building that kind of benchmark, which this article calls a done-means-done suite. It sets out the research question, the task record, the verification checks, the population and holdouts, the baselines, the uncertainty method and a pre-registration checklist. It reports no results, no model scores and no measured completion rates. Anyone who runs the study should publish the outcome with its own population, dates, denominators and limits.
The research question is narrow on purpose: when an agent attempts a realistic business task in a sandboxed copy of the relevant systems, how often is the intended business result actually present afterward, how often does the agent claim success when it is not, and what unintended changes does it leave behind? Each of those three quantities needs its own check, and none of them can be read from the transcript alone.
Why is an agent's own report of completion not a measurement?
A completion report is a claim made by the system under test, so scoring it would let the examinee grade itself. The useful quantity is the gap between the claim and the verified result, which is why the protocol records the claim separately and never uses it as evidence of success.
Published audits of agent benchmarks show how easily a check can drift away from the result it is meant to confirm. The Agentic Benchmark Checklist paper by Zhu and colleagues separates task validity, meaning a task can only be solved with the intended capability, from outcome validity, meaning the check really reflects success. Among its examples, an incorrect patch could still pass a coding benchmark's tests, and in tau-bench's airline tasks an agent that did nothing at all passed 38 percent of them. A business benchmark built on loose checks will report the same kind of inflated completion.
Crossing the agent's claim with the verified result, and splitting verified results by whether a forbidden change also occurred, gives five outcomes, each answering a different buyer question. The table below defines them; the false completion row is the one most deployments care about, because it is the case where a person believes work is finished when it is not.
| Agent claim | Verified result | Cell name | What it tells a deploying team |
|---|---|---|---|
| Says done | Business result present, no forbidden side effects | Verified completion | The task can be trusted to this agent at the measured rate |
| Says done | Result absent, partial or wrong | False completion | Someone will rely on work that does not exist; needs a downstream check or human review |
| Says done | Result present but a forbidden change also occurred | Completion with damage | The result is there but the agent broke something else, such as an extra refund or an overwritten field |
| Says not done or asks for help | Result absent | Honest stop | The agent knows its limits; usually acceptable if escalation is fast |
| Says not done | Result actually present | Under-report | Harmless for accuracy but wastes review time; often a sign of a weak self-check |
What should the artifact and state verification task suite include?
Each task in the suite is one record with a fixed set of fields, written and frozen before any model run. The fields below are the core of the asset. Example values are illustrative and come from a fictional accounts payable workflow, not from any real company.
The two fields that most often get skipped are the forbidden side effects and the claim capture. Without the first, an agent that posts the right invoice and also voids an unrelated one scores as a success. Without the second, the study cannot measure false completion at all.
| Field | What it holds | Illustrative value (accounts payable task) | Written by |
|---|---|---|---|
| Task id and family | Stable identifier, task family, source company code, version and freeze date | AP-0412, invoice exception handling, company C07, v1, frozen before runs | Benchmark lead |
| Business result statement | One or two sentences naming the outcome a manager would accept as done | The duplicate vendor invoice is voided, the valid one is approved for payment, and the vendor is told why | Domain expert |
| Initial state snapshot | De-identified sandbox data the task starts from, with a checksum | Ledger sandbox with two open invoices for one vendor, one a duplicate | Data engineer |
| Required artifacts | Files or messages that must exist afterward, with field-level checks | A vendor email that references both invoice numbers and the void reason | Domain expert |
| State assertions | Queries over the system of record that must return the target values | Duplicate invoice status equals voided; valid invoice status equals approved; amount unchanged | Benchmark lead and domain expert |
| Forbidden side effects | Changes that must not occur, checked by diffing the full end state | No other invoice changes status; no vendor master edits; no payment released | Domain expert |
| Acceptable variants | Other end states that also count as done, written as equivalence rules | Approval may be recorded by the agent or routed to an approver queue | Domain expert |
| Claim capture | Where and how the agent's own completion statement is recorded and coded | Final message parsed into done, not done or needs help | Benchmark lead |
| Delayed check | A downstream condition read after a simulated time step, where the workflow has one | Payment run picks up only the valid invoice | Data engineer |
| Rights and release status | Whether the source record is cleared for evaluation use and for publication of the task | Cleared for evaluation; task text not publishable | Rights reviewer |
Which checks inspect artifacts, and which inspect system state?
Artifact checks inspect things the agent produced, such as a spreadsheet, a document or a message, while state checks inspect the systems the agent changed. Most business tasks need both, plus a full-state diff for side effects.
Public agent benchmarks already show the mechanics. In the tau2-bench task schema, the default reward for its airline, retail and telecom domains multiplies a database check by a communication check: the predicted environment's end state is compared by hash with a target state produced by replaying one reference trajectory on a fresh environment, and any trajectory that reaches an equivalent end state passes. OSWorld's evaluators compare result files with golden files; its table metrics, for example, check a result spreadsheet against an expected one using sequential rules, and some score partial credit by aligning rows on a key column. SWE-bench pairs tests that must change from failing to passing with tests that must keep passing, which is the software version of a required result plus a forbidden side effect.
Business systems add two complications. The same result can be reached through different screens or APIs, so checks should read state, not replay clicks. And some results only become visible later, such as a payment run or a customer reply, so the suite needs a simulated time step for those tasks or must mark them as not verifiable inside the benchmark.
| Check type | What it inspects | Illustrative business example | Failure it catches |
|---|---|---|---|
| State assertion | Specific fields in the system of record after the run | Ticket status, resolution code and refund amount in a help desk sandbox | Agent says the refund was issued but the record never changed |
| Full-state diff | Every table or object the task could touch, compared with the target | All CRM contacts and deals before and after a merge task | Correct merge plus an unrelated contact deleted |
| Artifact content check | Fields inside a produced file, compared with a reference by rule | A reconciliation spreadsheet's totals and matched line items | File exists but the totals are copied from the wrong period |
| Communication check | Required facts present in a message to the simulated customer or colleague | Reply states the new appointment window and technician name | State is right but the customer was never told |
| Delayed or downstream check | A condition read after a simulated process runs | Scheduled job, payment batch or reorder trigger picks up the change | Change saved in a field the downstream process ignores |
| Claim coding | The agent's final statement, coded without seeing the result | Final message classified as done, not done or needs help | Needed to compute false completion; not a success check itself |
How do you turn a closed business record into a verifiable task?
Start from records whose business outcome was recorded and confirmed, then rebuild the starting state, not the ending one. A closed support ticket, a paid invoice or a completed work order tells you what a correct end state looked like; the task gives the agent the situation before that work happened.
Three rules keep the conversion honest. First, the historical outcome is one acceptable answer, not the only one; a domain expert writes equivalence rules for other end states a manager would accept, the same way tau2-bench treats its listed actions as one reference path rather than a requirement. Second, the snapshot is de-identified and frozen with a checksum before any model sees it, and the grading code and rubrics are hidden from the examinee; TheAgentCompany's evaluation guide, for example, lets benchmark users inspect checkpoint rubrics but does not allow agents to read rubrics or evaluation code. Third, the task is cleared for evaluation use by whoever controls the records before it enters the suite.
Records that recorded only activity, such as a ticket closed by automation with no resolution code, cannot anchor a verified-outcome task. They can still supply realistic starting states, but the expected result must then be written and reviewed by an expert rather than read from history, and the task should be flagged as expert-defined in the release notes.
Which companies, task families and holdouts make the result transferable?
A result transfers only if the tasks that score it were not drawn from the same companies and task families used to build or tune the agent. The protocol therefore holds out whole companies and whole task families, not just individual tasks.
Define the population as task families observed across several source companies, for example invoice exceptions, refund requests, appointment changes and CRM merges, each drawn from at least a few independent businesses. Sample tasks within each family by stratum, such as routine, exception and multi-system cases, and record the collection window for every source record so later readers can see how current the tasks are.
Split three ways. A development set is used to write and debug checks. A company holdout contains tasks from businesses that never contributed to development. A family holdout contains one or more task families that appear nowhere else. The pre-registered primary result is the score on the company holdout; the family holdout answers the separate question of whether verified completion carries over to work the builders did not anticipate. Report denominators for every cell, including tasks dropped because a check could not run.
What sample, baseline and uncertainty would make a result defensible?
A result is defensible when it reports verified completion and false completion against fixed baselines, with uncertainty that reflects both repeated trials and the clustering of tasks within companies. Point estimates alone are not enough.
At a glance
What sample, baseline and uncertainty would make a result defensible
Use at least three baselines. A null agent that takes no action and always claims success sets the floor and exposes checks that pass without work. A scripted reference that performs the documented steps confirms every task is solvable in the sandbox. A human reference, where practical, shows what a trained operator achieves under the same time and tool limits. Record the exact model name, version, date and agent configuration for every system tested.
Run every task several times. The tau-bench paper introduced pass^k, the chance that all k independent trials of a task succeed, to measure consistency rather than a best attempt; business deployments usually care about that version. Compute confidence intervals by resampling at the company level rather than the task level, because tasks from one business share systems and habits. Size the study from a pilot: measure the variance of the paired difference you care about, then choose the number of companies, tasks and trials that can detect the smallest difference you would act on.
Report these metrics, defined before any run:
- Verified completion rate: share of attempts where all state assertions, artifact checks and communication checks pass and no forbidden side effect occurs.
- Claim precision: among attempts the agent reported as done, the share that were verified complete.
- False completion rate: among all attempts, the share where the agent reported done and verification failed.
- Side-effect rate: share of attempts with at least one forbidden change, whether or not the main result was achieved.
- pass^k at the pre-registered k, by task family, with company-clustered intervals.
- Check coverage: share of tasks where every planned check executed, so missing checks are not silently counted as passes or failures.
How do you audit the graders before trusting a single score?
Graders are part of the measurement, so they get tested before any model does. A grader that accepts a wrong end state inflates verified completion, and one that rejects valid variants makes good agents look like they are failing.
Audit each task with planted solutions. Feed the grader the scripted correct end state and every documented acceptable variant; all must pass. Then feed it known-bad end states written by a domain expert, such as the right status on the wrong record, the right record with an extra change, or a correct file with one wrong total; all must fail. Tasks whose grader fails either test are fixed or removed before the freeze, and the removal is logged.
Use rule-based checks wherever the system of record can answer the question. Reserve model-based judges for residual judgments such as whether a message is clear, calibrate them on a human-labeled sample, and report their agreement. Finally, have a second expert spot-check a random sample of passed and failed attempts after the runs, blind to which model produced them, and publish the disagreement rate alongside the main result.
Pre-registration checklist for a done-means-done benchmark
File this checklist with a public registry or an internal dated record before the first model run. The Center for Open Science describes preregistration as specifying a research plan in advance and submitting it to a registry, which separates confirmatory analysis from exploration. Anything decided after seeing results is reported as exploratory.
At a glance
Pre-registration checklist for a done-means-done benchmark
Checklist · 12 items
- Research question and the three primary quantities: verified completion, false completion and side-effect rate.
- Task families, source companies, collection windows and the stratification rule.
- Development, company holdout and family holdout membership, frozen with checksums.
- The task record fields, including business result statements, state assertions, forbidden side effects and acceptable variants.
- Grader audit results with planted correct and planted wrong end states, and the list of removed tasks.
- Claim coding rules and who codes claims, blind to verification results.
- Baselines: null agent, scripted reference and human reference where practical.
- Systems under test with exact versions, dates, tool access and time limits.
- Number of trials per task, the pre-registered k, and the company-level resampling method for intervals.
- Smallest difference of interest and the pilot used to size the study.
- Exclusion rules for tasks where a check fails to run, and how they are reported.
- Release conditions: which tasks stay private, what is published, and the rights clearance behind each source record.
0 of 12 ticked. Tick each item as you go.
Requesting records to build a verified-outcome task suite
The hardest input to this protocol is not software but records: closed business cases whose outcomes were confirmed and whose owners have cleared them for evaluation. Teams that need such records from outside their own company should specify the task families, the systems involved, the outcome fields required for state assertions, the volume per family and whether the tasks must stay private.
Those requirements map directly onto a SourceX data request, which asks for scope, volume, language, format, timing and licensing needs; SourceX then searches U.S. businesses for matching closed cases instead of drawing on stock. The business supplying those cases approves scope and terms and signs ahead of delivery, and its signature fixes evaluation use and any publication limits. Neither step replaces the grader audit or the holdout design above; they only determine which records are cleared to enter the suite.
Frequently asked questions
What is a business task completion benchmark?
It is a test suite in which an AI agent attempts realistic business tasks in sandboxed copies of real systems, and success is judged by whether the intended business result exists afterward. The best versions check system state, produced files and required messages, and they also look for unintended changes. A score based on the agent's own report of completion does not count.
How is this different from grading an agent's transcript?
A transcript shows what the agent said and tried; it does not prove the ledger, ticket or calendar changed. State and artifact checks read the systems directly, so two different action paths that reach the same correct end state both pass, and a confident but empty claim fails.
What is false completion and why measure it separately?
False completion is an attempt where the agent reports the task as done but verification shows it is not. It matters more than a plain failure rate because people downstream act on the claim. Measuring it requires recording the claim and the verified result independently for every attempt.
Can historical business records serve as the answer key?
Partly. A confirmed outcome shows one acceptable end state, but a domain expert should add rules for other end states a manager would accept. Records that only show activity, with no confirmed outcome, need an expert-written expected result instead.
Which result would hold up on companies the builders never saw?
Only a score measured on a pre-registered holdout of whole companies, and ideally whole task families, that played no part in writing checks or tuning agents. Scores on development tasks usually overstate transfer, so report the holdout as the primary result.
Does this page report benchmark results?
No. It is a method only. Any team that runs the protocol should publish its results with the population, collection dates, denominators, baselines, uncertainty and grader audit described here.
Sources
- The tau2-bench task schema documentation says the default reward basis for its airline, retail and telecom domains is a database check multiplied by a communication check, and that any trajectory producing an equivalent database end state passes the database check. Source
- TheAgentCompany's evaluation guide describes grading checkpoints from the result and an optional trajectory used for partial credit, and says examinees are not allowed to read checkpoint rubrics or evaluation code. Source
- OSWorld's table evaluators compare a result spreadsheet or CSV with expected files by sequential rules, including a partial-credit comparison that aligns rows by a key column. Source
- The SWE-bench dataset guide lists FAIL_TO_PASS and PASS_TO_PASS test fields for each task instance, separating tests the fix must make pass from tests that must keep passing. Source
- Zhu et al. propose the Agentic Benchmark Checklist covering task validity, outcome validity and reporting, with examples including an incorrect patch passing tests and a do-nothing agent passing 38 percent of tau-bench airline tasks. Source
- The tau-bench paper compares the database state at the end of a conversation with an annotated goal state and introduces pass^k to measure whether an agent succeeds in all k independent trials. Source
- The Center for Open Science describes preregistration as specifying a research plan in advance of a study and submitting it to a registry, separating confirmatory from exploratory research. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.