AI uses for records
Invoice exception benchmark design: a protocol for rights-cleared cases
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
This is a protocol, not a result. To build a rights-cleared invoice exception benchmark, collect resolved accounts payable exceptions only under licenses that allow benchmark publication, de-identify them, turn each into a task with the invoice, purchase order, receipts and policy, and grade the agent's resolution against the recorded outcome with code checks first and calibrated expert review second. Hold out whole companies, publish a benchmark card, and report no numbers until grading is verified.
Evaluation split
Invoice exception benchmark
Key takeaways
- This page publishes a method only; it reports no results, scores or model rankings.
- Each task is one resolved exception: the invoice, purchase order, receipts, vendor record and policy go in, and the recorded resolution is the answer key.
- Grade the resolution action and corrected fields with code first, then the rationale with expert-calibrated review.
- Hold out whole companies and ERP families, so a result shows generalization rather than memorized vendor patterns.
- Rights come first: every contributing company must license benchmark use and publication, and de-identification alone does not grant that right.
How can you build a rights-cleared invoice exception benchmark?
You build it in five steps: license the cases, define the exception families, convert each resolved case into a task, write a grader that checks the resolution against what actually happened, and publish a benchmark card before any result. This page is the protocol for that work. It reports no measured findings, and any number on it is labeled illustrative.
The research question is narrow on purpose: given the documents an accounts payable specialist had at the time, does an AI system choose the same resolution the business recorded, change the right fields, and leave a hold in place when it should stay? That is the decision that costs money when it is wrong, and it is checkable against records that already exist.
The benchmark is open in the sense that its method, card and a public split are released. A private holdout stays with the maintainers so the public split can be used without destroying the benchmark's value.
What counts as an invoice exception in this benchmark?
An invoice exception is an invoice that the AP system stops from posting or paying because it fails a matching or control rule. ERP documentation describes the core cases. Oracle's Payables User's Guide describes matching holds such as Qty Rec, when the billed quantity exceeds the received quantity beyond tolerance, and Price, when the invoice unit price exceeds the purchase order price beyond tolerance; once the cause is corrected, resubmitting validation releases the hold, and some holds, such as Final Matching, cannot be released manually. Microsoft's Dynamics 365 Finance documentation describes three-way matching of invoice price to purchase order price and invoice quantity to product receipt quantity, configurable price tolerances, and a setting that can require approval before posting an invoice with matching discrepancies.
The benchmark groups cases into families so results can be reported per family. The first three below follow the documented matching rules; the others are common AP control exceptions that a contributing company's own policy must define.
| Exception family | Trigger | Resolution evidence in the record | Possible gold actions |
|---|---|---|---|
| Price variance | Invoice unit price above the PO price beyond tolerance | Hold history, PO revision, buyer note, credit memo | Release after PO change; request credit; reject |
| Quantity variance | Quantity billed exceeds quantity received | Later receipt, receiving correction, short-ship note | Wait for receipt; short-pay; release after receipt |
| Closed or final-matched PO | Invoice matched to a permanently closed PO | Reversed match, rematch to a new PO | Reverse match and rematch; reject |
| Missing PO or receipt | No PO reference or no receipt recorded | Requester confirmation, retroactive PO, receipt entry | Route to requester; hold |
| Suspected duplicate | Same vendor, amount and date or invoice number as a prior invoice | Void, cancellation or confirmation of a distinct charge | Cancel duplicate; release as distinct |
| Vendor master change | Remittance or bank details changed shortly before payment | Callback verification record, approval of change | Hold pending verification; release after verification |
Which exceptions go in, and why hold out whole companies?
The population is resolved invoice exceptions from contributing companies, where resolved means the record shows the final action and the invoice reached a terminal state: paid, credited, canceled or rejected. Open exceptions are excluded, because their answer is not yet known.
Sample by stratum, not at random. Stratify by exception family, by ERP family and by company, and cap any single company's share so one large contributor does not define the benchmark. Within a stratum, sample uniformly over time so seasonal patterns and policy changes are represented.
Hold out whole companies, not random cases. A result that holds only for cases from companies the system has seen may reflect memorized vendor names, part numbers or approver habits. The private holdout should contain at least one company per ERP family that appears nowhere in the public split, plus a later time slice from public-split companies to test drift.
Record the denominator: how many exceptions each company had in the collection window, how many met the inclusion rules, and how many were dropped at rights review or de-identification. Without it, readers cannot judge what the sample represents.
The benchmark card: what it must state before release
Publish the card with the protocol, before any results. It follows the lifecycle structure that Datasheets for Datasets proposed, motivation, composition, collection, uses and maintenance, adapted to a task benchmark. Entries in the third column are illustrative placeholders.
At a glance
The benchmark card: what it must state before release
| Card field | What it states | Illustrative entry |
|---|---|---|
| Purpose and question | The decision the benchmark measures | Choose the recorded resolution for an AP exception from the documents available at the time |
| Population and window | Who contributed, ERP families, collection dates | Mid-sized distributors and manufacturers; two ERP families; a named 24-month window |
| Sampling and denominator | Strata, caps, inclusion counts and drops | Per-family counts, company cap, cases dropped at rights review |
| Splits | Public split, private holdout, rules for each | Holdout of whole companies plus a later time slice |
| Task schema version | Schema fields and version tag | Task schema v1.0 with change log |
| Grader and version | Grader layers, code version, rubric version | State checker v1.0; rationale rubric v1.0 calibrated on a stated sample |
| Baselines | Rule baseline, human baseline, models with versions | Tolerance-rule baseline; AP specialist subset; pinned model identifiers |
| Rights basis | License terms allowing benchmark use and publication | Per-contributor license reference; publication scope |
| De-identification | What was removed, transformed or synthesized | Vendor names and bank details replaced; amounts kept; dates shifted consistently |
| Limits and maintenance | Known gaps, refresh cadence, contact for errors | No tax-specific exceptions; annual refresh; errata process |
The task schema for one exception case
Each task is one resolved exception turned into a self-contained packet. The rule is simple: everything the specialist could see at the time goes in the context, and everything that happened afterward goes in the answer key, never the other way round.
| Field | Content | Grader use |
|---|---|---|
| case_id and split | Stable pseudonymous ID; public or holdout | Joins results to the card; enforces split rules |
| exception_family and erp_family | One family from the table above; ERP family label | Per-family and per-ERP reporting |
| context_packet | Invoice, PO and revisions, receipts, vendor record, prior invoices as of the hold date | Input only; nothing dated after the hold |
| policy_excerpt | The contributor's tolerance and approval rules in force at the time | Lets graders judge policy compliance |
| allowed_actions | Release, hold, short-pay, request credit, reject, route to named role | Constrains the action space |
| gold_resolution | Recorded action and the corrected field values | Deterministic state check |
| gold_rationale_tags | Reasons the specialist recorded, as tags | Rubric check of the explanation |
| severity_if_wrong | Cost class of a wrong action, such as paying a duplicate | Severity-weighted scoring |
| rights_record | License reference and permitted uses | Release gate; excluded from model input |
| canary | Benchmark canary string | Contamination filtering and detection |
How should a grader check an AP resolution?
Grade in layers, cheapest and most objective first. The state layer copies the approach of agent benchmarks such as tau-bench, whose code compares the database state after a run with the state produced by replaying the annotated correct actions. Here, the agent's proposed action and field changes are applied to a copy of the task's records and compared with the gold resolution.
Anthropic's guidance on agent evaluation describes code-based, model-based and human graders, and notes that model-based graders need calibration against human judgment. The plan uses all three, each for the part it does best.
| Layer | What it checks | Method | Pass condition |
|---|---|---|---|
| 1. Action | Chosen action matches the gold action | Exact match on the allowed action list | Same action; alternatives only if listed in the card as equivalent |
| 2. State | Corrected fields after applying the action | Apply to a copy of the records and diff with gold | No difference in graded fields; no change to others |
| 3. Rationale | Explanation cites the right documents and reasons | Rubric scored by a model grader calibrated on expert labels | Required rationale tags present; no invented facts |
| 4. Adjudication | Disagreements and all severe errors | Two AP specialists, blind to the system's identity | Recorded decision; rubric updated if the gold was wrong |
Baselines, metrics and uncertainty
Three baselines make any future result interpretable. A rule baseline applies the contributor's tolerance rules mechanically; if an AI system does not beat it, the benchmark is measuring arithmetic. A human baseline has AP specialists solve a subset of tasks blind, which bounds achievable accuracy and exposes ambiguous gold labels. Model baselines record exact model identifiers, prompts and dates.
At a glance
Baselines, metrics and uncertainty
The primary metric is resolution accuracy per exception family: layers 1 and 2 both pass. Secondary metrics are the false release rate, releasing a hold that the record shows should have stayed, which is the costly error, and escalation precision. Report each with a Wilson score interval, which the NIST Engineering Statistics Handbook describes for proportions, and resample by company rather than by case, because cases from one company are not independent.
Before running anything, pre-register the protocol. The checklist below is what the registration should contain.
- Research question and the exact decision being graded.
- Population, collection window, strata, caps and the denominator table.
- Split rules, holdout companies and the time slice.
- Task schema and grader versions, with the rubric and equivalence rules.
- Baselines and pinned model identifiers.
- Primary and secondary metrics, interval method and resampling unit.
- Severity classes and how severe errors are reported.
- Exclusion rules decided in advance, and how dropped tasks are disclosed.
- Conditions for releasing results, and what happens if grading fails calibration.
Rights, privacy and release conditions
Four permissions are separate and all are needed. Access to the AP system does not mean the company may share the records; privacy rules govern what personal data in them may be used; ownership of the records may be clear while vendor contracts or confidentiality terms still restrict disclosure; and a license must state that benchmark construction and public release of the de-identified split are permitted uses. De-identification reduces privacy risk but does not by itself create a right to license or publish.
De-identify with a recorded method. NIST SP 800-188 describes removing direct identifiers, transforming quasi-identifiers and generating synthetic data, under governance such as a disclosure review board. For invoices, vendor names, bank details, addresses and approver names are direct identifiers; unusual amounts, part numbers and dates can act as quasi-identifiers. Shift dates consistently within a case so tolerance and timing logic still works.
Mark every released file with a canary. BIG-bench asks that task files and any external publication carry its canary string so benchmark data can be filtered out of training corpora; this benchmark should do the same with its own identifier. Release results only when grading has passed calibration, the holdout is intact, and legal and privacy reviewers have signed off on the public split.
What would a publishable invoice exception result have to show?
A result is worth publishing only if it holds on held-out companies and on exception families scored separately. A pooled accuracy that hides a weak duplicate or vendor-change family is not defensible, because those are the families where a wrong release causes the most harm.
A defensible result names the sample size per family, the interval, the baselines it beats, the grader version and calibration, and the drops. It also reports where the human baseline disagreed with the recorded gold, because some recorded resolutions were themselves mistakes.
Stop and narrow the scope if a family has too few cases for a useful interval, if specialists disagree often on the gold label, or if rights review removes one ERP family entirely. A smaller benchmark with clean labels and a clear card is more useful than a broad one with unverified grading.
Where rights-cleared cases come from
The cases sit in the AP systems of companies that process enough exceptions to have a history: distributors, manufacturers and service businesses with purchase-order-driven buying. Benchmark builders need them with hold history, notes and final actions intact.
Benchmark builders who need cases beyond their own company can ask SourceX to look for them among U.S. businesses; each search starts from a request rather than an existing collection. The request covers scope, volume, format, timing and licensing needs, and it should say plainly that building and publishing a benchmark must be a permitted use. A contributing company's AP records stay with it until it has signed off on the scope and terms of its agreement.
When should an invoice that matches its purchase order still not be paid?
When something that authorizes payment is missing, as opposed to something that checks arithmetic: the goods were not accepted, the vendor is on a payment hold, a dispute is open on the line, the approver lacked authority, or a contract condition has not been met. A three-way match shows that price and quantity agree across the purchase order, receipt and invoice. It does not show that the business agreed to pay.
Public rules and ERP documentation draw the same line. The Federal Acquisition Regulation at 32.905 (text through Federal Acquisition Circular 2025-06) bases payment on a proper invoice and satisfactory contract performance, and requires a receiving report that states the quantities received and accepted and the date of acceptance. Oracle Payables lists a Quality hold for an invoice whose billed quantity is more than the quantity accepted, a gap that a received-quantity match cannot see. In the Dynamics 365 Supplier Engagement app, a preview feature documented on July 27, 2026, a Payment hold on a vendor stops new payments from being generated. Odoo's 3-way matching sets a new bill's Should Be Paid field to Yes, because a bill can only be created once products have been received, and the documentation notes the status can be changed by hand.
Add these cases to the benchmark as a seventh family: matched but not payable. Each is a resolved invoice that passed matching within tolerance and whose recorded resolution was a hold, short payment, routing or rejection. The condition matrix below is the evidence packet for the family. Its extra documents go into context_packet, and the condition that blocked payment becomes a gold_rationale_tag.
| Blocking condition | Evidence added to the packet | What matching shows | Gold action when proven | Safe action when evidence is missing or conflicts |
|---|---|---|---|---|
| Received but not accepted | Inspection or acceptance record; rejection or return-to-vendor note | Billed quantity equals received quantity | Hold until acceptance; short-pay or request credit for rejected units | Hold and ask receiving or quality for the acceptance record |
| Vendor on payment hold | Vendor master hold type, reason and start date as of the decision | Nothing: the hold sits outside the three documents | Leave unpaid and cite the hold reason | Hold and route to the vendor master owner; never release on the agent's own reading |
| Open dispute on the line | Dispute log, vendor correspondence, pending credit memo request | A clean match on the original line | Hold, or pay the undisputed part where policy allows it | Hold the whole invoice and route to the dispute owner |
| Approval missing or beyond authority | Approval history and the delegation limits in force on the approval date | Nothing: approvals sit outside the three documents | Route to an approver with enough authority | Hold; an approval with no recorded authority counts as missing |
| Contract condition unmet | Contract clause, milestone sign-off, retention terms | Price and quantity agree with the purchase order | Hold, or pay net of retention as the contract states | Hold and route to the contract owner with the clause cited |
What should the agent do when the payment evidence is missing or contradicts itself?
Hold and ask for the specific record; never recommend payment to fill a gap. Release is the one action in the allowed list that is hard to undo, so the rule for this family has three parts: recommend payment only when every blocking condition has positive evidence dated on or before the decision; when an item is absent, hold and name the missing record and the role that owns it; when two sources disagree, such as a released hold in the vendor master and an open dispute in the log, escalate with both records cited rather than trusting the newer one.
Qualified accounts payable reviewers sign each case's gold action and its acceptable alternatives before any system runs. Anthropic's engineering post of January 9, 2026 counts a task as sound only if two domain experts, each deciding alone, would give it the same pass or fail verdict, so cases where the two reviewers split go to adjudication or leave the family; they are never scored as they stand. The reviewer-approved action rubric sorts every recommendation into one of five outcomes:
- Correct: the gold action, or an alternative the benchmark card lists as equivalent, with the blocking condition named.
- Correct but unsupported: the right action citing the wrong condition or none, scored as a failure at the page's rationale layer.
- Invalid payment recommendation: pay or release on a case whose gold is hold, route, short-pay or reject. Always in the most severe class.
- Unnecessary escalation: hold or route on a case whose gold is pay now. A cost to the business, not a control failure.
- Wrong route: the right decision to hold, sent to a role that cannot clear it, such as asking the requester to lift a vendor payment hold.
How do you compare a matching-only baseline with a context-aware decision test?
Run the same cases through two arms and report two error rates, never one blended accuracy. Arm A sees only the invoice, purchase order and receipt. Arm B adds the packet from the condition matrix. The page's rule baseline and pinned model baselines run in both arms, so the gap between arms measures what the extra evidence is worth.
The family must also contain matched invoices that should be paid, drawn from the same companies and period; otherwise a system that holds everything wins. Anthropic's post makes the general case: test both where a behavior should occur and where it should not, because one-sided evals create one-sided optimization.
Arm A cannot see the blocking evidence, so its invalid payment rate on this family mainly shows how often a system pays on a clean match. The readings that matter are arm B's two rates below, and whether a system that improves one gives it back on the other. No cases have been collected or graded, so this addition, like the protocol above it, reports no results.
Count four things per arm, each over its own denominator:
- Invalid payment recommendation rate: pay or release recommendations on gold-hold cases, divided by all gold-hold cases in the family.
- Unnecessary escalation rate: hold or route recommendations on gold-pay cases, divided by all gold-pay cases in the family.
- Consistency: the share of gold-hold cases with no invalid payment in any of k repeated runs, which matches the post's pass^k, the probability that all k trials succeed.
- Arm gap: the difference in each rate between arm A and arm B, with an interval resampled by company as the protocol already requires.
Sources for this section
- FAR 32.905 (GSA's FAR source, text through FAC 2025-06) bases payment on a proper invoice and satisfactory contract performance and requires a receiving report stating quantities received and accepted and the date of acceptance. Source
- Oracle Payables (Release 12.1 reference guide) lists a Quality hold for an invoice whose billed quantity is more than the quantity accepted, released after acceptance is entered and validation is resubmitted. Source
- Microsoft's Dynamics 365 Supplier Engagement documentation (preview, July 27, 2026) says a Payment hold on a vendor stops new payments from being generated, while existing payments can still post. Source
- Odoo's 3-way matching documentation says a new vendor bill's Should Be Paid field is set to Yes because a bill cannot be created until some products are received, and that the status can be changed manually. Source
- Anthropic's engineering post (January 9, 2026) says a good task is one where two domain experts would independently reach the same verdict, recommends testing cases where a behavior should and should not occur, and defines pass^k. Source
Frequently asked questions
Does this page report benchmark results?
No. It is a protocol: the card, task schema, grader plan, sampling and release rules for a benchmark that has not been run. Any example on the page is illustrative, and results should be published only after grading has been calibrated and the holdout is intact.
What should the benchmark card, task schema and grader plan include?
The card states the question, population, collection window, sampling and denominator, splits, schema and grader versions, baselines, rights basis, de-identification method and limits. The task schema separates the context available at the time from the recorded resolution. The grader plan checks action and state with code, the rationale with a calibrated rubric, and disagreements with two specialists.
Why hold out whole companies instead of random cases?
Cases from one company share vendors, approvers, tolerances and writing habits, so random splits let a system score well by recognizing patterns it has already seen. Holding out whole companies, and a later time slice, tests whether the system generalizes to a business it has never encountered.
Can de-identified invoices be published without a license?
Not safely. De-identification reduces privacy risk, but the records may still be confidential under vendor contracts or company policy, and the company must agree to benchmark use and publication. Counsel should review the license terms and the public split before release.
What sample size makes a result defensible?
Enough cases per exception family that the interval is narrow enough to separate the systems or baselines being compared. Report a Wilson interval per family, resample by company, and state the count of tasks rather than the count of repeated runs.
Sources
- Oracle's Payables User's Guide (Release 12.1) describes Qty Rec and Price matching holds, release of holds by resubmitting validation after the cause is corrected, and holds such as Final Matching that do not allow manual release. Source
- Dynamics 365 Finance three-way matching compares invoice price with PO price and invoice quantity with receipt quantity, with price tolerances and optional approval before posting discrepancies. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
- tau-bench compares the post-run database state with the state produced by the annotated correct actions. Source
- Agent evaluations combine code-based, model-based and human graders; model-based graders need calibration against humans. Source
- BIG-bench asks that task files and external publications include its canary string to keep benchmark data out of training corpora. Source
- NIST SP 800-188 describes de-identification techniques and governance for datasets. Source
- The NIST handbook describes the Wilson score interval for a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.