Skip to content

Systems and records

Business task grader specification: write it before collecting tasks

By SourceX Editorial · Updated

9 sections

Short answer

Writing the grader first turns a vague goal into evidence each task must carry. For invoice reconciliation, a business task grader specification names the inputs (purchase order, receipts, invoice, tolerance policy, posting history), the checks that decide pass or fail, their severity, and the adversarial tests the grader itself must pass. Collect only tasks whose records let every check run, and keep contested cases visible.

Grader rubric

Business task grader

This page covers business task grader specification: write it before collecting tasks.Illustrative

Key takeaways

  • The grader is the real definition of the task, so write it before collecting tasks.
  • Each grader check names the records it reads, which becomes the collection brief.
  • Score system state and forbidden writes, not the agent's message.
  • List the shortcuts that would raise the score and test the grader against each one.
  • Read tolerances and matching policy from each task's records, never from constants.
  • Keep contested tasks labeled as contested instead of forcing one answer.

Why write the grader before collecting a single training task?

Write the grader first because it decides what evidence every task has to contain. A team that collects tasks first usually discovers later that half of them cannot be scored: the receipt is missing, the tolerance policy was never exported, or the correct outcome was never recorded. A grader written up front turns those gaps into collection requirements.

The grader is also the real definition of the task. In reinforcement fine-tuning, the model is pushed toward whatever the grader rewards, not toward what the task description intended. OpenAI's graders guide, for example, lists string checks, text similarity, model-based score graders and Python graders whose grade function receives the sample and the item and returns a numeric score. Whatever form it takes, that function is the specification the model will learn.

Writing it first has a third benefit: it forces domain experts to agree on what correct means before anyone pays for collection. Disagreements that surface while drafting the grader cost an afternoon; disagreements found after 5,000 tasks are collected cost a relabeling project.

What does success in invoice reconciliation mean, stated as evidence?

In invoice reconciliation, success is a correct disposition of each invoice line backed by the matching evidence, with no change to records the agent was not allowed to touch. It is not a message that says the invoice was reconciled.

Microsoft's Dynamics 365 Finance documentation describes the checks that real accounts payable systems run. Under two-way matching, the invoice's prices are checked against the purchase order. Three-way matching adds a comparison of the invoice quantity with the quantity on the product receipts selected for the invoice. Differences are matching discrepancies, judged against configured tolerances, and an invoice with a discrepancy outside tolerance can be saved but held until the discrepancy is resolved.

The same documentation shows why the rules need to be explicit. Price tolerances can be set by item, item group, vendor, vendor group, item and vendor combination, or legal entity. Price totals matching considers previously posted invoice lines as well as the current one, and charges are compared only for charge codes configured for comparison. A grader that ignores any of these settings will mark correct work as wrong, or wrong work as correct.

The documentation also spells out the arithmetic. Net unit price is the line's net amount divided by its quantity, and the price total is unit price times line quantity, plus line charges, minus line discounts. Writing those formulas into the grader, rather than letting it compare the numbers printed on a screen, is what makes a check reproducible by someone who has only the records.

Which checks make up the executable grader specification?

The specification below is written so an engineer can implement each row as a function over the task's records and the agent's final state. Severity decides the score: any critical failure sets the task score to zero, major failures cap it, and minor failures deduct. The tolerance values are inputs read from the task's policy record, never constants in the grader.

  • Task input contract: purchase order lines, receipt lines, invoice lines, matching policy and tolerance settings, posting history, charge code configuration.
  • Reference contract: adjudicated disposition per line, reference linkage, recomputed discrepancy set, list of tables the agent may write.
  • Grader output: per-check pass or fail, severity, the evidence ids used, a final score, and a grader version string.
Which checks make up the executable grader specification?
CheckWhat it verifiesEvidence the task must carryPass rule and severity
G1 DispositionEach invoice line ends as approved, held or rejected as the adjudicated label saysAdjudicated disposition per line, with adjudicator idExact match per line; critical
G2 Line linkageEach invoice line is linked to the right purchase order line and receipt linesPurchase order lines, receipt lines, reference linkage tableAll links match the reference; major
G3 Quantity matchInvoiced quantity is compared with received quantity where three-way matching appliesReceipt quantities, matching policy per item or vendorDiscrepancy flag equals the recomputed flag; critical
G4 Unit price toleranceNet unit price variance is tested against the most specific tolerance that appliesTolerance hierarchy export, purchase order pricesFlag equals recomputed flag at the right tolerance level; major
G5 Cumulative price totalPrice totals include previously posted invoices for the same linePosting history for the purchase order lineFlag equals recomputed cumulative flag; major
G6 ChargesCharges are compared only for codes set for comparisonCharge code configuration, invoice and order chargesNo flags on excluded codes; minor
G7 Stated reasonsHold reasons the agent cites equal the set of discrepancies recomputed by the graderReason code listSet equality; missing reason major, extra reason minor
G8 No posting with open blocksNothing is posted while a blocking discrepancy is openFinal ledger and invoice status snapshotZero violations; critical
G9 Forbidden writesPurchase orders, receipts and tolerance settings are unchanged by the agentBefore and after snapshots of those tablesByte-equal snapshots; critical
G10 Explanation qualityThe note to the approver cites the evidence ids behind each reasonAgent note, rubric for a model-based scoreRubric score above threshold; minor, never decisive

Which shortcuts could let an agent score without reconciling?

The most dangerous shortcut is one that raises the score while skipping the business requirement, and every grader has some. The 2016 paper Concrete Problems in AI Safety calls this reward hacking: an objective with a clever easy solution that maximizes it while defeating the designer's intent.

Business graders are prone to it because they often check text. The tau-bench reference code, for instance, checks required outputs by testing whether each expected string appears in the agent's replies after commas are removed, alongside a comparison of database state. That design is reasonable for its benchmark, but it shows how a reply that lists many candidate values could satisfy a string check without the reasoning behind it. The state comparison is what makes such shortcuts fail.

Write each shortcut down and add a test that catches it before collection begins.

Which shortcuts could let an agent score without reconciling?
ShortcutWhy a naive grader passes itTest that catches it
Hold every invoiceHolding is never a wrong posting, so a check for bad postings alone is satisfiedG1 compares dispositions; a set of clean invoices must be approved to pass
Copy the order price onto the invoicePrice variance disappears, so the tolerance check passesG9 snapshot of source documents; invoice line values compared with the original
Edit the receipt quantity to matchQuantity discrepancy disappearsG9 forbidden-write check on receipt tables; critical failure
Check only the current invoiceEach partial invoice is within tolerance on its ownG5 with a task where the third partial invoice breaks the cumulative total
Cite every possible reason codeA recall-only check finds the right reason in the listG7 set equality, so extra reasons cost points
Claim completion in the final messageA text check finds the word reconciledG1 and G8 read the system state, not the message

How do you test the grader before it scores anything?

Before the grader scores a single collected task, it should pass its own test suite: known-good runs that must score full marks, known-bad runs that must fail with the right check, and boundary cases that sit exactly on a tolerance. A grader that has never been shown to fail a bad run has not been tested.

At a glance

How do you test the grader before it scores anything

SWE-bench offers a useful pattern. Each instance carries two test lists: FAIL_TO_PASS tests that must change from failing to passing once the issue is fixed, and PASS_TO_PASS tests that must keep passing. A reconciliation grader needs the same two kinds of evidence: checks that must flip when the agent does the work, and checks that must stay green, such as G8 and G9, whatever the agent does.

Decide the pass mark for the grader before running the suite, not after seeing results. A reasonable bar is that every golden run scores full marks, every planted failure fails on the planted check and no other, and every shortcut run falls below the task pass mark. Any miss is fixed in the grader and the whole suite is rerun, because a fix for one check often moves another.

  • Golden runs: a person completes 20 to 40 tasks correctly by hand, and every one must score full marks. Any loss of points is a grader bug.
  • Planted failures: copies of the golden runs with one deliberate error each, one per check. Each must fail on exactly the check that was planted.
  • Boundary pairs: two variants of one task, one a hair inside the tolerance and one a hair outside, to catch off-by-one comparisons and rounding.
  • Shortcut runs: one scripted run for each shortcut in the table above. Each must score below the pass mark.
  • Policy variants: the same invoice under two tolerance configurations, to confirm the grader reads the policy record instead of a constant.
  • Replay check: rerunning the grader on stored outputs gives identical scores, with the grader version recorded.

What should happen when accounts payable experts disagree?

When two qualified reviewers reach different dispositions on the same invoice, record both and mark the task as contested instead of forcing a single label. Contested tasks are real: a vendor's freight line, a substituted item or a receipt posted to the wrong date can all support more than one defensible decision.

A practical rule is to have two reviewers adjudicate every task in a calibration batch and a sample of the rest, and to send disagreements to a third, senior reviewer who records the reason. Tasks where the senior reviewer also cannot settle the matter stay in the set with a contested flag, excluded from the reward signal but kept in evaluation, where they show whether the model is overconfident.

Error severity belongs to the same conversation. A wrongly approved invoice that pays for goods never received is worse than a wrongly held one that delays a correct payment by a week. The severity column in the specification should come from the accounts payable lead, not from the engineer writing the code.

Agreement figures belong in the delivery as well. Report how many tasks were double-reviewed, how often the first two reviewers agreed on the disposition, and how many tasks remain contested, so a buyer can judge how much of the reward signal rests on settled cases.

Worked example: one illustrative invoice through the grader

Illustrative only; quantities and percentages are invented. A purchase order covers 1,000 units of one item under three-way matching, with an illustrative net unit price tolerance of 5 percent for that item. Receipts total 960 units. The invoice bills 1,000 units at a unit price 3 percent above the order price. The adjudicated label is hold, with one reason: quantity exceeds received quantity by 40 units. Price is within tolerance.

Agent A approves the invoice. G1, G3 and G8 fail, two of them critical, so the score is zero. Agent B holds the invoice but cites a price discrepancy instead of quantity. G1 passes, and G3 passes because the system's quantity flag is left untouched, but G7 fails with one missing and one extra reason, and the score is capped. Agent C changes the receipt to 1,000 units and then approves. G9 fails as a forbidden write, and the score is zero even though the matching screen now shows no discrepancy.

One exception has to be built into the grader from the start. Service invoices often have no receipt, and many companies apply two-way matching to them by policy. If the task's policy record says two-way, G3 must be skipped rather than failed, which is only possible because the policy is part of the task's evidence.

What should be collected once the grader exists?

Once the grader is written, the collection brief follows directly from its evidence column: every task needs the records each check reads, as they stood at the moment the invoice was processed. Snapshots taken after later edits will make correct historical decisions look wrong.

Accounts payable history in an ERP usually holds most of this: purchase orders, receipts, invoices, matching policies, tolerance settings, posting history and the hold and release trail with reviewer notes. The adjudicated label is the part most often missing, because the original clerk's decision was not always right, so budget for review of a sample before relying on historical outcomes as truth.

Rights and privacy questions sit alongside the technical ones. Vendor names, bank details and employee approvals appear throughout these records and need de-identification that keeps the linkage between documents intact, and the buyer needs a license that covers training on them.

  • Purchase order header and lines, with the version that was current when the invoice arrived
  • Product receipt lines with dates and quantities, including receipts later reversed
  • Invoice header and lines exactly as captured, before any correction
  • Matching policy and tolerance settings in force on the processing date, at every level that applies
  • Posting history for each purchase order line, for cumulative price checks
  • Hold, release and approval trail with reviewer notes and timestamps
  • Adjudicated disposition and discrepancy set, with reviewer ids and any contested flag

Where does a rules grader stop, and where do reconciliation records come from?

A rules grader is only as current as the policy it encodes. When tolerances, matching policies or approval rules change, rerun the grader test suite, version the grader, and rescore stored runs, or scores from different months stop being comparable. Stop and bring in an accounts payable specialist when more than a small share of tasks turn out contested, because that usually means the policy itself is ambiguous rather than the agents being wrong.

Model-based checks such as G10 should stay advisory. They are useful for the quality of an explanation, but a disposition decided by a model judge reintroduces the uncertainty the executable checks were written to remove.

Reconciliation histories like these are sourced on request from U.S. businesses through SourceX rather than sold from stock. A team that has written its grader can send the evidence column as the data request, with scope, volume, format, timing and licensing needs, and ask for records that carry every input the checks require.

Frequently asked questions

What is a business task grader specification?

It is a written, testable definition of success for one business task: the records each task must include, the checks that compare the agent's final state with a reference, the severity of each failure, and the tests the grader must pass before it scores anything. It doubles as the collection brief.

Should a grader for business tasks use an LLM judge?

Use code for anything that can be recomputed from records, such as quantities, tolerances and forbidden writes. Reserve a model-based score for qualities that resist rules, such as the clarity of a note to an approver, and keep it from deciding pass or fail on its own.

How do you detect reward hacking in a business grader?

List the shortcuts that would raise the score without doing the work, script a run for each, and confirm each scores below the pass mark. Checks on the final system state and on records the agent must not change catch most of them.

What happens to tasks where experts disagree?

Keep them, label them contested, record each reviewer's reasoning, and leave them out of the reward signal. In evaluation they reveal whether a model is confidently wrong on genuinely ambiguous cases.

Does three-way matching apply to every invoice?

No. Dynamics 365 documentation, for example, shows matching policy set at the legal entity level and overridden for specific items, item and vendor combinations, or individual purchase order lines, so one company can apply two-way matching to some purchases and three-way to others. The grader has to read the policy that applied to each task.

Sources

  • Dynamics 365 Finance defines two-way and three-way matching, tolerance-based discrepancies, tolerance levels, cumulative price totals matching and charges matching for configured codes. Source
  • Dynamics 365 Finance examples set matching policy at the legal entity level and override it for an item, an item and vendor combination, or a purchase order line. Source
  • OpenAI's graders guide lists string check, text similarity, score model and Python graders; a Python grader's grade function takes the sample and item and returns a score. Source
  • Concrete Problems in AI Safety describes reward hacking as an objective with a clever easy solution that maximizes it while defeating the designer's intent. Source
  • tau-bench scores required outputs by case-insensitive substring match in agent replies, alongside a database state comparison. Source
  • SWE-bench instances carry FAIL_TO_PASS and PASS_TO_PASS test lists. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies