Skip to content

AI uses for records

Invoice exception benchmark design: a protocol for rights-cleared cases

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

This is a protocol, not a result. To build a rights-cleared invoice exception benchmark, collect resolved accounts payable exceptions only under licenses that allow benchmark publication, de-identify them, turn each into a task with the invoice, purchase order, receipts and policy, and grade the agent's resolution against the recorded outcome with code checks first and calibrated expert review second. Hold out whole companies, publish a benchmark card, and report no numbers until grading is verified.

Evaluation split

Invoice exception benchmark

This page covers invoice exception benchmark design: a protocol for rights-cleared cases.Illustrative

Key takeaways

  • This page publishes a method only; it reports no results, scores or model rankings.
  • Each task is one resolved exception: the invoice, purchase order, receipts, vendor record and policy go in, and the recorded resolution is the answer key.
  • Grade the resolution action and corrected fields with code first, then the rationale with expert-calibrated review.
  • Hold out whole companies and ERP families, so a result shows generalization rather than memorized vendor patterns.
  • Rights come first: every contributing company must license benchmark use and publication, and de-identification alone does not grant that right.

How can you build a rights-cleared invoice exception benchmark?

You build it in five steps: license the cases, define the exception families, convert each resolved case into a task, write a grader that checks the resolution against what actually happened, and publish a benchmark card before any result. This page is the protocol for that work. It reports no measured findings, and any number on it is labeled illustrative.

The research question is narrow on purpose: given the documents an accounts payable specialist had at the time, does an AI system choose the same resolution the business recorded, change the right fields, and leave a hold in place when it should stay? That is the decision that costs money when it is wrong, and it is checkable against records that already exist.

The benchmark is open in the sense that its method, card and a public split are released. A private holdout stays with the maintainers so the public split can be used without destroying the benchmark's value.

What counts as an invoice exception in this benchmark?

An invoice exception is an invoice that the AP system stops from posting or paying because it fails a matching or control rule. ERP documentation describes the core cases. Oracle's Payables User's Guide describes matching holds such as Qty Rec, when the billed quantity exceeds the received quantity beyond tolerance, and Price, when the invoice unit price exceeds the purchase order price beyond tolerance; once the cause is corrected, resubmitting validation releases the hold, and some holds, such as Final Matching, cannot be released manually. Microsoft's Dynamics 365 Finance documentation describes three-way matching of invoice price to purchase order price and invoice quantity to product receipt quantity, configurable price tolerances, and a setting that can require approval before posting an invoice with matching discrepancies.

The benchmark groups cases into families so results can be reported per family. The first three below follow the documented matching rules; the others are common AP control exceptions that a contributing company's own policy must define.

What counts as an invoice exception in this benchmark?
Exception familyTriggerResolution evidence in the recordPossible gold actions
Price varianceInvoice unit price above the PO price beyond toleranceHold history, PO revision, buyer note, credit memoRelease after PO change; request credit; reject
Quantity varianceQuantity billed exceeds quantity receivedLater receipt, receiving correction, short-ship noteWait for receipt; short-pay; release after receipt
Closed or final-matched POInvoice matched to a permanently closed POReversed match, rematch to a new POReverse match and rematch; reject
Missing PO or receiptNo PO reference or no receipt recordedRequester confirmation, retroactive PO, receipt entryRoute to requester; hold
Suspected duplicateSame vendor, amount and date or invoice number as a prior invoiceVoid, cancellation or confirmation of a distinct chargeCancel duplicate; release as distinct
Vendor master changeRemittance or bank details changed shortly before paymentCallback verification record, approval of changeHold pending verification; release after verification

Which exceptions go in, and why hold out whole companies?

The population is resolved invoice exceptions from contributing companies, where resolved means the record shows the final action and the invoice reached a terminal state: paid, credited, canceled or rejected. Open exceptions are excluded, because their answer is not yet known.

Sample by stratum, not at random. Stratify by exception family, by ERP family and by company, and cap any single company's share so one large contributor does not define the benchmark. Within a stratum, sample uniformly over time so seasonal patterns and policy changes are represented.

Hold out whole companies, not random cases. A result that holds only for cases from companies the system has seen may reflect memorized vendor names, part numbers or approver habits. The private holdout should contain at least one company per ERP family that appears nowhere in the public split, plus a later time slice from public-split companies to test drift.

Record the denominator: how many exceptions each company had in the collection window, how many met the inclusion rules, and how many were dropped at rights review or de-identification. Without it, readers cannot judge what the sample represents.

The benchmark card: what it must state before release

Publish the card with the protocol, before any results. It follows the lifecycle structure that Datasheets for Datasets proposed, motivation, composition, collection, uses and maintenance, adapted to a task benchmark. Entries in the third column are illustrative placeholders.

At a glance

The benchmark card: what it must state before release

The benchmark card: what it must state before release
Card fieldWhat it statesIllustrative entry
Purpose and questionThe decision the benchmark measuresChoose the recorded resolution for an AP exception from the documents available at the time
Population and windowWho contributed, ERP families, collection datesMid-sized distributors and manufacturers; two ERP families; a named 24-month window
Sampling and denominatorStrata, caps, inclusion counts and dropsPer-family counts, company cap, cases dropped at rights review
SplitsPublic split, private holdout, rules for eachHoldout of whole companies plus a later time slice
Task schema versionSchema fields and version tagTask schema v1.0 with change log
Grader and versionGrader layers, code version, rubric versionState checker v1.0; rationale rubric v1.0 calibrated on a stated sample
BaselinesRule baseline, human baseline, models with versionsTolerance-rule baseline; AP specialist subset; pinned model identifiers
Rights basisLicense terms allowing benchmark use and publicationPer-contributor license reference; publication scope
De-identificationWhat was removed, transformed or synthesizedVendor names and bank details replaced; amounts kept; dates shifted consistently
Limits and maintenanceKnown gaps, refresh cadence, contact for errorsNo tax-specific exceptions; annual refresh; errata process

The task schema for one exception case

Each task is one resolved exception turned into a self-contained packet. The rule is simple: everything the specialist could see at the time goes in the context, and everything that happened afterward goes in the answer key, never the other way round.

The task schema for one exception case
FieldContentGrader use
case_id and splitStable pseudonymous ID; public or holdoutJoins results to the card; enforces split rules
exception_family and erp_familyOne family from the table above; ERP family labelPer-family and per-ERP reporting
context_packetInvoice, PO and revisions, receipts, vendor record, prior invoices as of the hold dateInput only; nothing dated after the hold
policy_excerptThe contributor's tolerance and approval rules in force at the timeLets graders judge policy compliance
allowed_actionsRelease, hold, short-pay, request credit, reject, route to named roleConstrains the action space
gold_resolutionRecorded action and the corrected field valuesDeterministic state check
gold_rationale_tagsReasons the specialist recorded, as tagsRubric check of the explanation
severity_if_wrongCost class of a wrong action, such as paying a duplicateSeverity-weighted scoring
rights_recordLicense reference and permitted usesRelease gate; excluded from model input
canaryBenchmark canary stringContamination filtering and detection

How should a grader check an AP resolution?

Grade in layers, cheapest and most objective first. The state layer copies the approach of agent benchmarks such as tau-bench, whose code compares the database state after a run with the state produced by replaying the annotated correct actions. Here, the agent's proposed action and field changes are applied to a copy of the task's records and compared with the gold resolution.

Anthropic's guidance on agent evaluation describes code-based, model-based and human graders, and notes that model-based graders need calibration against human judgment. The plan uses all three, each for the part it does best.

How should a grader check an AP resolution?
LayerWhat it checksMethodPass condition
1. ActionChosen action matches the gold actionExact match on the allowed action listSame action; alternatives only if listed in the card as equivalent
2. StateCorrected fields after applying the actionApply to a copy of the records and diff with goldNo difference in graded fields; no change to others
3. RationaleExplanation cites the right documents and reasonsRubric scored by a model grader calibrated on expert labelsRequired rationale tags present; no invented facts
4. AdjudicationDisagreements and all severe errorsTwo AP specialists, blind to the system's identityRecorded decision; rubric updated if the gold was wrong

Baselines, metrics and uncertainty

Three baselines make any future result interpretable. A rule baseline applies the contributor's tolerance rules mechanically; if an AI system does not beat it, the benchmark is measuring arithmetic. A human baseline has AP specialists solve a subset of tasks blind, which bounds achievable accuracy and exposes ambiguous gold labels. Model baselines record exact model identifiers, prompts and dates.

At a glance

Baselines, metrics and uncertainty

The primary metric is resolution accuracy per exception family: layers 1 and 2 both pass. Secondary metrics are the false release rate, releasing a hold that the record shows should have stayed, which is the costly error, and escalation precision. Report each with a Wilson score interval, which the NIST Engineering Statistics Handbook describes for proportions, and resample by company rather than by case, because cases from one company are not independent.

Before running anything, pre-register the protocol. The checklist below is what the registration should contain.

  • Research question and the exact decision being graded.
  • Population, collection window, strata, caps and the denominator table.
  • Split rules, holdout companies and the time slice.
  • Task schema and grader versions, with the rubric and equivalence rules.
  • Baselines and pinned model identifiers.
  • Primary and secondary metrics, interval method and resampling unit.
  • Severity classes and how severe errors are reported.
  • Exclusion rules decided in advance, and how dropped tasks are disclosed.
  • Conditions for releasing results, and what happens if grading fails calibration.

Rights, privacy and release conditions

Four permissions are separate and all are needed. Access to the AP system does not mean the company may share the records; privacy rules govern what personal data in them may be used; ownership of the records may be clear while vendor contracts or confidentiality terms still restrict disclosure; and a license must state that benchmark construction and public release of the de-identified split are permitted uses. De-identification reduces privacy risk but does not by itself create a right to license or publish.

De-identify with a recorded method. NIST SP 800-188 describes removing direct identifiers, transforming quasi-identifiers and generating synthetic data, under governance such as a disclosure review board. For invoices, vendor names, bank details, addresses and approver names are direct identifiers; unusual amounts, part numbers and dates can act as quasi-identifiers. Shift dates consistently within a case so tolerance and timing logic still works.

Mark every released file with a canary. BIG-bench asks that task files and any external publication carry its canary string so benchmark data can be filtered out of training corpora; this benchmark should do the same with its own identifier. Release results only when grading has passed calibration, the holdout is intact, and legal and privacy reviewers have signed off on the public split.

What would a publishable invoice exception result have to show?

A result is worth publishing only if it holds on held-out companies and on exception families scored separately. A pooled accuracy that hides a weak duplicate or vendor-change family is not defensible, because those are the families where a wrong release causes the most harm.

A defensible result names the sample size per family, the interval, the baselines it beats, the grader version and calibration, and the drops. It also reports where the human baseline disagreed with the recorded gold, because some recorded resolutions were themselves mistakes.

Stop and narrow the scope if a family has too few cases for a useful interval, if specialists disagree often on the gold label, or if rights review removes one ERP family entirely. A smaller benchmark with clean labels and a clear card is more useful than a broad one with unverified grading.

Where rights-cleared cases come from

The cases sit in the AP systems of companies that process enough exceptions to have a history: distributors, manufacturers and service businesses with purchase-order-driven buying. Benchmark builders need them with hold history, notes and final actions intact.

Benchmark builders who need cases beyond their own company can ask SourceX to look for them among U.S. businesses; each search starts from a request rather than an existing collection. The request covers scope, volume, format, timing and licensing needs, and it should say plainly that building and publishing a benchmark must be a permitted use. A contributing company's AP records stay with it until it has signed off on the scope and terms of its agreement.

When should an invoice that matches its purchase order still not be paid?

When something that authorizes payment is missing, as opposed to something that checks arithmetic: the goods were not accepted, the vendor is on a payment hold, a dispute is open on the line, the approver lacked authority, or a contract condition has not been met. A three-way match shows that price and quantity agree across the purchase order, receipt and invoice. It does not show that the business agreed to pay.

Public rules and ERP documentation draw the same line. The Federal Acquisition Regulation at 32.905 (text through Federal Acquisition Circular 2025-06) bases payment on a proper invoice and satisfactory contract performance, and requires a receiving report that states the quantities received and accepted and the date of acceptance. Oracle Payables lists a Quality hold for an invoice whose billed quantity is more than the quantity accepted, a gap that a received-quantity match cannot see. In the Dynamics 365 Supplier Engagement app, a preview feature documented on July 27, 2026, a Payment hold on a vendor stops new payments from being generated. Odoo's 3-way matching sets a new bill's Should Be Paid field to Yes, because a bill can only be created once products have been received, and the documentation notes the status can be changed by hand.

Add these cases to the benchmark as a seventh family: matched but not payable. Each is a resolved invoice that passed matching within tolerance and whose recorded resolution was a hold, short payment, routing or rejection. The condition matrix below is the evidence packet for the family. Its extra documents go into context_packet, and the condition that blocked payment becomes a gold_rationale_tag.

When should an invoice that matches its purchase order still not be paid?
Blocking conditionEvidence added to the packetWhat matching showsGold action when provenSafe action when evidence is missing or conflicts
Received but not acceptedInspection or acceptance record; rejection or return-to-vendor noteBilled quantity equals received quantityHold until acceptance; short-pay or request credit for rejected unitsHold and ask receiving or quality for the acceptance record
Vendor on payment holdVendor master hold type, reason and start date as of the decisionNothing: the hold sits outside the three documentsLeave unpaid and cite the hold reasonHold and route to the vendor master owner; never release on the agent's own reading
Open dispute on the lineDispute log, vendor correspondence, pending credit memo requestA clean match on the original lineHold, or pay the undisputed part where policy allows itHold the whole invoice and route to the dispute owner
Approval missing or beyond authorityApproval history and the delegation limits in force on the approval dateNothing: approvals sit outside the three documentsRoute to an approver with enough authorityHold; an approval with no recorded authority counts as missing
Contract condition unmetContract clause, milestone sign-off, retention termsPrice and quantity agree with the purchase orderHold, or pay net of retention as the contract statesHold and route to the contract owner with the clause cited

What should the agent do when the payment evidence is missing or contradicts itself?

Hold and ask for the specific record; never recommend payment to fill a gap. Release is the one action in the allowed list that is hard to undo, so the rule for this family has three parts: recommend payment only when every blocking condition has positive evidence dated on or before the decision; when an item is absent, hold and name the missing record and the role that owns it; when two sources disagree, such as a released hold in the vendor master and an open dispute in the log, escalate with both records cited rather than trusting the newer one.

Qualified accounts payable reviewers sign each case's gold action and its acceptable alternatives before any system runs. Anthropic's engineering post of January 9, 2026 counts a task as sound only if two domain experts, each deciding alone, would give it the same pass or fail verdict, so cases where the two reviewers split go to adjudication or leave the family; they are never scored as they stand. The reviewer-approved action rubric sorts every recommendation into one of five outcomes:

  • Correct: the gold action, or an alternative the benchmark card lists as equivalent, with the blocking condition named.
  • Correct but unsupported: the right action citing the wrong condition or none, scored as a failure at the page's rationale layer.
  • Invalid payment recommendation: pay or release on a case whose gold is hold, route, short-pay or reject. Always in the most severe class.
  • Unnecessary escalation: hold or route on a case whose gold is pay now. A cost to the business, not a control failure.
  • Wrong route: the right decision to hold, sent to a role that cannot clear it, such as asking the requester to lift a vendor payment hold.

How do you compare a matching-only baseline with a context-aware decision test?

Run the same cases through two arms and report two error rates, never one blended accuracy. Arm A sees only the invoice, purchase order and receipt. Arm B adds the packet from the condition matrix. The page's rule baseline and pinned model baselines run in both arms, so the gap between arms measures what the extra evidence is worth.

The family must also contain matched invoices that should be paid, drawn from the same companies and period; otherwise a system that holds everything wins. Anthropic's post makes the general case: test both where a behavior should occur and where it should not, because one-sided evals create one-sided optimization.

Arm A cannot see the blocking evidence, so its invalid payment rate on this family mainly shows how often a system pays on a clean match. The readings that matter are arm B's two rates below, and whether a system that improves one gives it back on the other. No cases have been collected or graded, so this addition, like the protocol above it, reports no results.

Count four things per arm, each over its own denominator:

  • Invalid payment recommendation rate: pay or release recommendations on gold-hold cases, divided by all gold-hold cases in the family.
  • Unnecessary escalation rate: hold or route recommendations on gold-pay cases, divided by all gold-pay cases in the family.
  • Consistency: the share of gold-hold cases with no invalid payment in any of k repeated runs, which matches the post's pass^k, the probability that all k trials succeed.
  • Arm gap: the difference in each rate between arm A and arm B, with an interval resampled by company as the protocol already requires.

Sources for this section

  • FAR 32.905 (GSA's FAR source, text through FAC 2025-06) bases payment on a proper invoice and satisfactory contract performance and requires a receiving report stating quantities received and accepted and the date of acceptance. Source
  • Oracle Payables (Release 12.1 reference guide) lists a Quality hold for an invoice whose billed quantity is more than the quantity accepted, released after acceptance is entered and validation is resubmitted. Source
  • Microsoft's Dynamics 365 Supplier Engagement documentation (preview, July 27, 2026) says a Payment hold on a vendor stops new payments from being generated, while existing payments can still post. Source
  • Odoo's 3-way matching documentation says a new vendor bill's Should Be Paid field is set to Yes because a bill cannot be created until some products are received, and that the status can be changed manually. Source
  • Anthropic's engineering post (January 9, 2026) says a good task is one where two domain experts would independently reach the same verdict, recommends testing cases where a behavior should and should not occur, and defines pass^k. Source

Frequently asked questions

Does this page report benchmark results?

No. It is a protocol: the card, task schema, grader plan, sampling and release rules for a benchmark that has not been run. Any example on the page is illustrative, and results should be published only after grading has been calibrated and the holdout is intact.

What should the benchmark card, task schema and grader plan include?

The card states the question, population, collection window, sampling and denominator, splits, schema and grader versions, baselines, rights basis, de-identification method and limits. The task schema separates the context available at the time from the recorded resolution. The grader plan checks action and state with code, the rationale with a calibrated rubric, and disagreements with two specialists.

Why hold out whole companies instead of random cases?

Cases from one company share vendors, approvers, tolerances and writing habits, so random splits let a system score well by recognizing patterns it has already seen. Holding out whole companies, and a later time slice, tests whether the system generalizes to a business it has never encountered.

Can de-identified invoices be published without a license?

Not safely. De-identification reduces privacy risk, but the records may still be confidential under vendor contracts or company policy, and the company must agree to benchmark use and publication. Counsel should review the license terms and the public split before release.

What sample size makes a result defensible?

Enough cases per exception family that the interval is narrow enough to separate the systems or baselines being compared. Report a Wilson interval per family, resample by company, and state the count of tasks rather than the count of repeated runs.

Sources

  • Oracle's Payables User's Guide (Release 12.1) describes Qty Rec and Price matching holds, release of holds by resubmitting validation after the cause is corrected, and holds such as Final Matching that do not allow manual release. Source
  • Dynamics 365 Finance three-way matching compares invoice price with PO price and invoice quantity with receipt quantity, with price tolerances and optional approval before posting discrepancies. Source
  • Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
  • tau-bench compares the post-run database state with the state produced by the annotated correct actions. Source
  • Agent evaluations combine code-based, model-based and human graders; model-based graders need calibration against humans. Source
  • BIG-bench asks that task files and external publications include its canary string to keep benchmark data out of training corpora. Source
  • NIST SP 800-188 describes de-identification techniques and governance for datasets. Source
  • The NIST handbook describes the Wilson score interval for a proportion. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies