AI uses for records
How to turn an AI agent pilot into evidence procurement can use
By SourceX Editorial · Updated
10 sections
Short answer
Turn an AI agent pilot into procurement evidence by fixing the workload and decision criteria first, then running every vendor on the same frozen set of real cases several times. Score completion against your system of record, grade every failure by business severity, report paired differences with intervals by task family, and list what the pilot did not test. Put it all in one signed comparison report.
Agent run
Agent evaluation procurement
Key takeaways
- Write the decision criteria and gates before any vendor result is seen, and date them.
- Build the case set from your own work volume by task family, oversampling rare families where one error is costly.
- Score completion from evidence in the system of record and grade each failure by business severity with a blind reviewer.
- Compare vendors case by case with intervals clustered by task family; an interval that includes zero is not a win.
- Every report names the task families and failure types it did not measure.
What evidence does procurement need from an AI agent pilot?
Procurement needs a report it can compare across vendors without trusting any vendor's demo: the target workload defined in business terms, a frozen evaluation set that every vendor ran unchanged, verified completion rates by task family, errors weighted by business severity, repeated trials and uncertainty ranges, and decision criteria written down before anyone saw a score.
Most pilots produce the opposite: a few weeks of vendor-chosen tasks, a satisfied sponsor and a slide of impressive transcripts. That is useful for learning, but a buyer cannot defend a contract award with it, and a losing vendor can fairly ask why its pilot was judged on different work.
Public-sector buyers are now told to test before they buy. OMB memorandum M-25-22, issued in April 2025, directs federal agencies to test proposed AI solutions during proposal evaluation where practicable so they understand what the systems can and cannot do, and it strongly encourages performance-based techniques for setting requirements and contract terms. Private buyers have no such mandate, but the same logic holds: the pilot is the only test you will run before signing.
Think of the report as the pilot's contract with itself. Once the workload, the case set, the scoring rules and the decision criteria are fixed in writing, every later argument about the result is an argument about evidence rather than about what the pilot was supposed to measure. That is what lets procurement, finance and the business owner sign the same recommendation.
Why does a successful pilot demo not settle a vendor decision?
Because a demo answers whether the agent can do the task once, while procurement must know how often it does the task correctly on your mix of work, and how badly it fails when it does not. Those are different numbers, and the gap between them is where contracts go wrong.
Repeat runs make the gap visible. The tau-bench authors introduced pass^k, the chance that an agent succeeds on all of k independent trials of the same task, and reported that even strong function-calling agents of the time solved fewer than half of the tasks and were markedly inconsistent across trials. An agent that succeeds on a task in one demo run may fail the same task on the next.
Graders need checking too. When OpenAI built SWE-bench Verified, it found that the unit tests used to judge solutions were often overly specific, so correct fixes could be marked wrong. If your pilot scores depend on a vendor's own grader, or on a quick rubric no one audited, a ranking can reflect the grader rather than the agent.
A third problem is selection. Vendors understandably steer pilots toward the cases their product handles well, and sponsors understandably remember the transcripts that impressed them. Neither is dishonest, but both bias the evidence in the same direction. The cure is a case set the buyer draws and freezes before any vendor touches it, with the vendors told the task families but never the cases.
How do you define the workload and check coverage before the pilot starts?
Start from your own work volume, not from the vendor's feature list. Pull a period of real cases from the system of record, group them into task families, and decide how many evaluation cases each family gets, deliberately oversampling rare families where a single error is expensive.
The coverage matrix below is the first page of the report. The shares and counts are illustrative, for a hypothetical accounts payable exception agent; replace them with your own.
Two rules keep the matrix honest. First, every family that appears in real volume appears as a row, even if it gets zero cases, so the gap is visible on page one rather than buried in an appendix. Second, the case count per family is set by the decision you need to make, not by the share of volume: a family that is 2 percent of volume but can cause a fraudulent payment may need as many cases as a family twenty times its size. Sample cases from a recent period the vendors have not seen and keep the case list away from vendor staff until all runs are complete.
| Task family | Share of real volume (illustrative) | Cases in eval set (illustrative) | Highest severity tier | Covered in pilot? |
|---|---|---|---|---|
| Price mismatch between invoice and purchase order | 38% | 60 | S2 | Yes |
| Quantity mismatch against goods receipt | 24% | 45 | S2 | Yes |
| Missing or invalid purchase order number | 17% | 40 | S1 | Yes |
| Duplicate invoice suspected | 9% | 40 | S3 | Yes |
| Vendor bank detail change on invoice | 2% | 30 | S3 | Yes, oversampled |
| Tax code or jurisdiction error | 6% | 25 | S2 | Partly: two of five jurisdictions |
| Credit memo application | 4% | 0 | S2 | No, listed as unmeasured |
How should completion and error severity be scored?
Score completion against evidence in the system of record, not against the agent's own account of what it did. For an AP agent that means the invoice status, hold codes and posted amounts after the run; for a support agent it means the ticket state and the actions logged. Then grade every failure by what it would cost the business, because a pilot where one vendor makes fewer but worse errors should not look better.
Agree the severity rubric with the business task owner before the pilot, and have a reviewer who does not know which vendor produced a run assign severity to every failure.
Handoffs deserve their own number. An agent that refuses hard cases can post a high accuracy on the cases it accepts while shifting the real work back to your team. Report the handoff rate per family next to completion, and check a sample of handoffs to confirm the stated reason was correct; a handoff with a wrong reason is a failure, not a safe one.
| Tier | Definition | Example in AP exceptions | How it enters the decision |
|---|---|---|---|
| S0 | Correct outcome with evidence in the record | Hold released after matching goods receipt | Counts toward completion rate |
| S1 | Safe failure: no action taken, handed to a person with a correct reason | Routes an unclear PO case to the AP queue | Counts as not completed; tracked as handoff rate |
| S2 | Wrong action that a normal control would catch | Applies the wrong tax code; caught at review | Reported per task family with rework time |
| S3 | Wrong action that could cause loss, fraud or a compliance breach | Approves payment to changed bank details without verification | Any confirmed S3 above the agreed ceiling fails the vendor |
What should the vendor comparison report with decision criteria include?
The report is a fixed template that every vendor's results drop into, signed by named owners. Write the decision criteria in section 2 before results arrive and do not change them afterward; if you must, record the change and the reason in section 9.
Separate gates from weights. Gates are pass or fail conditions any vendor must meet, such as a ceiling on S3 errors or a minimum completion rate on the highest-volume family. Weighted criteria then rank the vendors that pass every gate: completion by family, handoff burden, consistency across trials, integration effort and operating cost. Keeping prices out of the evaluation file and adding them at the award stage prevents cost from leaking into how failures are graded.
| Report section | What it contains | Signed by | Red flag |
|---|---|---|---|
| 1. Target workload | Business process, systems, period sampled, volume and task families | Business task owner | Workload described in the vendor's terms, not yours |
| 2. Decision criteria | Gates (minimum completion, S3 ceiling) and weighted criteria, written before results | Procurement lead and task owner | Criteria dated after the first results |
| 3. Coverage matrix | Families, shares, case counts, oversampled families and unmeasured families | Evaluation lead | No row marked unmeasured |
| 4. Test conditions | Frozen case set, identical access and data, number of trials, harness version | Evaluation lead | Vendors ran different case subsets |
| 5. Results by family | Completion, handoff rate, S2 and S3 counts per family and vendor | Evaluation lead | Only a single blended score |
| 6. Uncertainty | Paired differences between vendors with intervals, clustered by family; consistency over repeated trials | Evaluation statistician | Point estimates without ranges |
| 7. Grader audit | Sample of verdicts re-checked by a blind human reviewer, agreement rate | Domain adjudicator | Vendor-supplied grader not audited |
| 8. Cost to operate | Run time, human review time per case and handoff workload (no prices in the evaluation file) | Operations owner | Review effort left out |
| 9. Limits and deviations | Unmeasured risks, changed criteria, incidents during the pilot | Procurement lead | Section left empty |
How do you show uncertainty so a close result is not called a win?
Run every vendor on the same cases and report the difference between vendors case by case, with an interval. If the interval for the difference includes zero, the report says the pilot could not separate them and the decision falls to other criteria.
Evan Miller's paper on adding error bars to evals recommends exactly this: when two systems answer the same questions, compute the standard error from the per-question differences, which is tighter than comparing two independent scores, and use clustered standard errors when questions come in related groups. In an agent pilot the clusters are your task families, because cases from one family tend to succeed or fail together.
Repeated trials belong in the same section. Run each case several times per vendor and report both the average success rate and the share of cases solved on every trial. The tau2-bench leaderboard, for example, asks submitters to run all tasks with the same configuration and strongly prefers at least four trials per domain. A vendor whose single-run rate is high but whose all-trials rate collapses has a reliability problem that a single demo would hide.
Write the uncertainty section for readers who are not statisticians. One sentence per comparison is enough: which vendor was ahead, by how much on the paired difference, the interval, and whether the interval excludes zero. Put the method details in an appendix that the evaluation statistician signs, so a reviewer can check the arithmetic without the main report turning into a methods paper.
Which important business task or failure type is absent from the evaluation?
Something always is, and the report must name it. The NIST AI Risk Management Framework asks that risks which will not or cannot be measured be documented, and that the effectiveness of the test and evaluation metrics themselves be evaluated. In a procurement report that becomes section 9: a list of what the pilot did not test and why.
Walk this list with the task owner before closing the report:
- Rare, high-severity families that had too few real cases to sample, such as fraud attempts or legal holds.
- Handoffs: what the agent does when it is unsure, and whether the human queue can absorb the volume.
- Inputs that were cleaned before the pilot but will not be in production, such as scanned documents or free-text email.
- Permission boundaries: actions the agent could take in production that the pilot sandbox blocked.
- Volume and timing: month-end spikes, batch loads and slow downstream systems.
- Policy changes during the contract term that will change what counts as correct.
Illustrative worked example: three vendors on an AP exception pilot
This example is illustrative; the vendors are unnamed and every count is invented to show the method, not reported results. A buyer freezes 240 cases across six covered families, runs three vendors three times each with identical sandbox access, and has a blind reviewer grade severity.
Vendor A completes the most cases on a single run but produces two S3 errors, both approving invoices with changed bank details; the gate in section 2 sets the S3 ceiling at zero, so A fails regardless of its completion rate. Vendor B and Vendor C pass the gate. B's completion is higher on price and quantity mismatches, C's on missing purchase orders. The paired difference between B and C across all cases has an interval that includes zero, so the report records that the pilot did not separate them on completion.
The weighted criteria then decide: C hands off more cases to people, which the operations owner estimates as more review hours per week; B solves more cases on all three trials. The report recommends B, notes credit memo handling as unmeasured, and makes a measured follow-up on that family a condition of the contract.
Notice what the report did not do: it did not average Vendor A's strong completion against its S3 errors into a single blended score, and it did not declare B the winner on completion alone. Each decision traces to a criterion written before the pilot, which is what makes the recommendation defensible when a losing vendor asks for a debrief.
When is a full comparison report overkill, and when do you need specialists?
A lighter version is fine for a low-risk internal tool with one vendor, no S3 failure modes and an easy exit: keep the coverage matrix, the severity rubric and the limits section, and skip the statistician. Use the full report whenever more than one vendor is in contention, the agent can move money or change records others rely on, or a losing bidder could challenge the award.
Bring in an evaluation statistician when the intervals are wide or families are small, a domain adjudicator when severity calls are disputed, and counsel or privacy staff before sending production records into any vendor's environment. If vendors cannot be given identical access, for example because one needs a different integration, say so in section 4 and treat their results as not directly comparable.
Keep the run records. Evaluation tools such as Inspect store each sample's input, output, target and score in the log, and logs like these are what let an auditor, or a vendor contesting the decision, re-check a verdict months later.
Finding enough real cases to fill the coverage matrix
The hardest rows to fill are the rare, expensive ones, and your own history may not hold enough of them. Licensed records from other companies running the same process can fill those cells, provided the license covers evaluation use and the records stay out of any vendor's training data.
Rare-family cases suit a SourceX request: it sources records from U.S. businesses for each buyer rather than selling stock, and its request form asks for the scope, volume, format, timing and licensing terms, which is where evaluation-only use should be written. No record reaches the buyer's pilot until the business that holds it has agreed the scope and terms and signed.
Frequently asked questions
What is an agent evaluation procurement report?
It is a fixed-format report that compares AI agent vendors on the buyer's own workload. It defines the task families, reports verified completion and severity-weighted errors from a frozen case set that every vendor ran, shows uncertainty, and applies decision criteria written before results were seen.
How many cases does an AI agent pilot need?
Enough per task family that the interval on the difference between vendors is narrow enough to act on, which depends on how close the vendors are. Oversample rare, high-severity families. If the interval still includes zero, report that the pilot could not separate the vendors rather than adding cases after the fact.
Should vendors grade their own pilot results?
No. Score completion against evidence in your system of record and have a blind reviewer assign severity. If a vendor's grader is used for speed, audit a sample of its verdicts with a human reviewer and report the agreement rate.
Why run each pilot case more than once?
Agents are not deterministic, and a case solved once may fail on the next run. Reporting the share of cases solved on every trial, alongside the average success rate, exposes reliability problems that a single demo run hides.
What if a pilot cannot test a critical task?
List it as unmeasured in the limits section, state why, and decide whether the contract should require a measured follow-up before the agent handles that task in production. Never let an untested family inherit the score of tested ones.
Sources
- OMB memorandum M-25-22 directs federal agencies to test proposed AI solutions during proposal evaluation where practicable and strongly encourages performance-based techniques for requirements and contract terms. Source
- The tau-bench paper defines pass^k as success on all k trials of a task and reports that strong function-calling agents solved fewer than half of tasks and were inconsistent across trials. Source
- OpenAI found that SWE-bench unit tests were often overly specific, so correct solutions could be graded as wrong. Source
- Miller recommends computing standard errors from question-level paired differences when comparing two models and clustered standard errors when questions come in related groups. Source
- The tau2-bench leaderboard asks submitters to run all tasks with a consistent configuration and strongly prefers at least four trials per domain. Source
- The NIST AI Risk Management Framework asks that risks that will not or cannot be measured be documented and that the effectiveness of TEVV metrics and processes be evaluated. Source
- Inspect evaluation logs store each evaluated sample with its input, output, target and score. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.