AI uses for records
What evidence proves an AI agent completed a business task?
By SourceX Editorial · Updated
10 sections
Short answer
An AI agent has completed a business task only when the system of record shows the required end state, nothing outside the task changed, every side effect happened exactly once, and the change is attributable to the run. The agent's final message is a claim to check, not evidence. Score claimed and verified completion separately and report false completion as its own metric.
Agent run
Agent evaluation completion
Key takeaways
- The final message is a claim; completion is the state of the system of record after the run.
- Verify four things per run: the required end state, no collateral changes, side effects exactly once, and an audit trail that ties the change to the run.
- Report claimed completion, verified completion and false completion separately; a false completion is usually worse than an honest failure.
- Weight failures by business severity and report verified completion per task family with an interval, not a single headline rate.
- Evaluation sets need real cases whose outcome the records already prove, so the expected end state is known before the agent runs.
What evidence proves an AI agent completed a business task?
Proof of completion is the state of the business system after the run, checked against the state the task required. For agent evaluation completion verification, that means reading the record the task was supposed to change, confirming the right fields hold the right values, confirming nothing else changed, and confirming any external action, such as a refund, an email or a ticket update, happened exactly once.
Anthropic's guidance on agent evaluations draws the same line. It calls the full record of a trial the transcript, and the final state of the environment the outcome. Its example is a flight-booking agent that says the flight has been booked; the outcome is whether a reservation exists in the environment's database, not what the agent said.
Everything else, including the agent's summary, its reasoning and its tool call log, is supporting material. It helps explain a failure, but it does not establish that the work is done.
Why is a confident final message not proof of completion?
A final message can be wrong in two directions, and only checking the record tells you which. An agent can report success after a tool call failed, after it updated the wrong record, or after it did half the work. It can also finish the task and then report it poorly, which matters less but still costs a person time.
Treat every run as two independent observations: what the agent claimed and what the evidence shows. Crossing them gives four outcomes, and each deserves a different response from the team that owns the agent.
| Agent claims | Evidence shows | Outcome name | Why it matters |
|---|---|---|---|
| Done | Done | Verified completion | The only outcome that counts toward the completion rate. |
| Done | Not done or partly done | False completion | The most expensive failure: people stop checking and the gap surfaces later as a customer or ledger problem. |
| Not done or unsure | Done | Silent completion | Work is correct but a person may redo it; track it as a reporting defect. |
| Not done or unsure | Not done | Honest failure | Cheap to catch; the case routes to a person with an accurate status. |
How do public agent benchmarks verify completion?
The best-known agent benchmarks grade the environment, not the message. Each one fixes in advance what evidence the grader will read, which is the habit worth copying for business tasks.
In the tau-bench code, the reward function replays the annotated ground-truth actions on a fresh copy of the data, hashes the result, and sets the reward to zero if the agent's database state does not match. When a task also requires information to be given to the user, it checks that the required outputs appear in the agent's responses.
WebArena's evaluation harness combines string matching on the answer, URL matching on where the agent ended up, and a program_html check that navigates to a page and reads specific content with a locator, so a task passes only if the site itself reflects the change. OSWorld task files carry an evaluator block that first runs post-configuration steps, such as saving the open spreadsheet, and then compares the resulting file with an expected file.
The lesson for a business evaluation is direct: before any agent runs, write down which record, file or page the grader will read, which fields it will compare, and what must not have changed.
The completion evidence checklist
Use these ten checks as the grading contract for each task family. Not every task needs all ten, but each one you drop should be a deliberate decision recorded in the task definition.
| Evidence item | What to check | Where it usually lives | Typical way it fails |
|---|---|---|---|
| 1. Target record | The run touched the intended record, matched by a stable key, not by name | CRM, ERP or ticket ID; order or invoice number | Agent updates a similarly named account or a duplicate record |
| 2. Required end state | Every required field holds the expected value after a settle window | Record fields read through the system API after the run | Status changed but the amount, date or owner did not |
| 3. No collateral change | A before and after diff shows changes only in allowed fields and records | Snapshot of the test tenant before and after | Agent edits unrelated fields or closes sibling tickets |
| 4. Side effects exactly once | Refunds, emails, orders and API writes occurred once | Payment, email and integration logs; idempotency keys | A retry creates a second refund or a duplicate message |
| 5. Attributable trail | An audit entry ties the change to the agent's identity and run | Field history or audit events, such as Zendesk ticket audits | Change exists but cannot be tied to the run, so it may predate it |
| 6. Policy constraints | Limits, approvals and segregation rules were respected | Policy text in the task packet; approval records | Correct end state reached by skipping a required approval |
| 7. Communication matches state | Any message to a customer or colleague agrees with the record | Sent email, ticket reply or chat message | Message promises a refund the ledger never shows |
| 8. Claim reconciliation | The agent's final status matches checks 1 to 7 | Final message parsed into a status field | Agent says done when checks fail, or unsure when they pass |
| 9. Downstream acceptance | The next step in the process accepts the result | Matching run, approval queue, reopen or reply within a window | Invoice posts but fails the next matching run; ticket reopens |
| 10. Evidence snapshot | All evidence is captured at a fixed time and stored with the run | Evaluation store keyed by run ID | Grader reads a live system that later runs have changed |
Which systems give you trustworthy completion evidence?
Prefer evidence the business system writes itself over evidence the agent produces. A ticket system's audit history is a strong source: Zendesk, for example, documents each ticket audit as a read-only record of one update, with events that carry the previous and new values of changed fields. An agent cannot easily fake that, and it lets you separate a change the run made from one that already existed.
For side effects, check the system that performs them. Payment APIs such as Stripe support idempotency keys so a client can retry a request without performing the same operation twice; in an evaluation, a duplicate charge, refund or message with a different key is direct evidence of a retry bug, whatever the agent reported.
Where no audit trail exists, take a full snapshot of the test environment before the run and diff it afterward. This is slower, but it is the only way to prove check 3, that nothing outside the task changed.
Run every evaluation in a test tenant or sandbox that is reset to the same starting state before each trial. A shared live environment lets one run's changes leak into the next run's evidence, so a later agent can appear to complete a task that an earlier run already finished. Record the reset version with each run so a reviewer can reproduce the grade.
Finally, decide the settle window per task family. Some systems apply changes asynchronously through queues or scheduled jobs, so reading the record one second after the agent stops can produce a false failure. Pick a window long enough for normal processing, document it, and read all evidence at that moment.
A worked scorecard: six illustrative runs
The example below is illustrative, not measured. It shows how one task family, a customer address change with a shipping fee refund, would be scored when each run is graded against the checklist instead of the final message.
| Run (illustrative) | Final message claim | Evidence found | Verdict | Severity |
|---|---|---|---|---|
| R1 | Address updated and refund issued | Checks 1 to 9 pass | Verified completion | None |
| R2 | Address updated and refund issued | Address updated; refund call returned an error and was not retried | False completion | S2 reversible business error |
| R3 | Address updated and refund issued | Refund issued twice under two different request keys | False completion | S3 financial harm |
| R4 | Could not confirm the refund | Address and refund both correct and logged once | Silent completion | S1 reporting defect |
| R5 | Address updated and refund issued | Checks 1 to 9 pass | Verified completion | None |
| R6 | Escalated: account on legal hold | No change made; hold flag present in the record | Honest failure, correct escalation | None |
How should you score severity and report uncertainty?
Report three rates per task family, not one. In the illustrative scorecard, claimed completion is 4 of 6 runs, verified completion is 3 of 6 including the silent completion, and false completion is 2 of the 4 runs that claimed success. The headline a vendor might quote, 4 of 6, hides that half of those claims were false.
Weight failures by what they cost the business. A simple four-level scale is enough to start: S0 no impact, S1 reporting or cosmetic defect, S2 reversible business error a person can fix, S3 financial, customer or compliance harm or an action that cannot be undone. Any S3 in a task family should block release for that family regardless of the overall rate.
Small task counts make rates noisy, so publish an interval. The NIST Engineering Statistics Handbook gives the Wilson method for a confidence interval on a proportion and notes that it is the interval recommended by Brown, Cai and DasGupta and by Agresti and Coull. Count tasks, not trials, as the sample size, and when consistency matters, report pass^k, the probability that all k repeated trials succeed, alongside pass@k, the probability that at least one does.
Anthropic's guidance also recommends grading the outcome rather than the path where possible, and calibrating any model-based grader against human graders. Use code checks for items 1 to 5 and 10, and reserve model or human review for items 6, 7 and 9, where judgment is needed.
Which business tasks or failure types is your evaluation missing?
Most completion evaluations over-sample clean cases and miss the failures that produce false completion. Build a coverage matrix with task families as rows and the failure types below as columns, and make sure every cell that is plausible in production has at least a few tasks.
- Partial completion: one of two required changes made, such as the address but not the refund.
- Wrong record: a duplicate customer, a sibling order or an archived ticket with a similar name.
- Duplicate side effect: a timeout followed by a retry that charges, refunds or emails twice.
- Policy bypass: the right end state reached without a required approval or above a limit.
- Stale read: the agent acts on data another process changed during the run.
- Correct refusal: the case should be escalated, as with a legal hold, and doing the task would be the failure.
- Silent completion: work done but reported as failed or uncertain.
When should a task family be rebuilt, shrunk or sent to a finance or system owner?
Stop and redesign when you cannot write the expected end state before the run. If the correct outcome depends on a judgment nobody recorded, the task measures the grader, not the agent; rebuild it from a case whose outcome the records already prove.
Narrow the scope when a task family has fewer than a few dozen tasks, when its interval is too wide to separate the agents you are comparing, or when one failure type dominates. A smaller, well-evidenced family is more useful than a broad one graded by reading final messages.
Ask for specialist review when severity depends on law, money movement or customer commitments. A finance owner should sign off on what counts as S3 for payment tasks, and the system owner should confirm that the audit fields you rely on are complete for the operations the agent performs.
Where do evaluation records with verifiable outcomes come from?
Completion checks need cases where the correct end state is already known: resolved tickets with their field history, orders with refund logs, invoices with their final posting. Teams usually start with their own systems and then look for comparable records from other companies to test generalization.
When your own systems cannot supply enough proven cases, SourceX finds licensed business records for AI teams on request from U.S. businesses and handles the rights review, contracting, preparation and delivery. A request spells out scope, volume, format, timing and licensing needs, and it is the place to name the audit and field-history columns your completion checks will read.
Frequently asked questions
What is the difference between task completion rate and verified completion rate?
A task completion rate is often computed from what the agent reports or from a grader reading its final message. A verified completion rate counts only runs where the system of record shows the required end state with no collateral changes. The gap between the two is the false completion rate, which should be reported separately.
Should an evaluation grade the agent's path or only the end state?
Grade the end state first, because several valid paths can reach it. Add path checks only where the path itself is a requirement, such as an approval step, a spending limit or a rule that a person must be consulted. Anthropic's evaluation guidance makes the same recommendation to grade what the agent produced rather than the path it took.
Can an LLM judge verify completion?
An LLM judge is useful for items that need judgment, such as whether a customer message is accurate and appropriate. It should not replace code checks of record state, side effects and audit entries, which are cheaper and deterministic. Calibrate any model grader against human reviewers on a sample before relying on it.
How many tasks do I need to report a completion rate?
Enough that the interval around the rate is narrow enough for your decision; with a few dozen tasks per family, intervals remain wide. Count distinct tasks rather than repeated trials, publish a Wilson interval, and report each task family on its own rather than pooling easy and hard work.
What should happen when the agent correctly refuses a task?
Score it as a success if the task was designed to require escalation, for example when a record carries a legal hold. Include such cases deliberately, because an agent that completes everything is not safe if some cases should never be completed automatically.
Sources
- Anthropic distinguishes the transcript of a trial from its outcome, the final state of the environment, describes code-based, model-based and human graders, and defines pass@k and pass^k. Source
- tau-bench sets the reward to zero when the post-run database state differs from the state produced by the annotated ground-truth actions, and checks required outputs in responses. Source
- WebArena grades tasks with string_match, url_match and program_html evaluators that read the resulting page content. Source
- OSWorld task files define an evaluator that runs post-configuration steps and compares the resulting file with an expected file. Source
- Zendesk ticket audits are read-only records of each ticket update, including field changes with previous and new values. Source
- Stripe idempotency keys let a client retry a request without performing the same operation twice. Source
- The NIST Engineering Statistics Handbook describes the Wilson score interval for a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.