Skip to content

AI uses for records

Grader executable invariants: reconciliation and permission checks

By SourceX Editorial · Updated

10 sections

Short answer

To turn business invariants into objective agent checks, write each rule as a predicate that code evaluates over the records before and after the run, and grade the end state rather than the transcript. Use reconciliation invariants for balances and matches, permission invariants for who may act, and a frame invariant that nothing else changed. Test the grader on shortcut runs before it scores agents.

Grader rubric

Grader executable invariants

This page covers grader executable invariants: reconciliation and permission checks.Illustrative

Key takeaways

  • An invariant is a rule the records must obey after the run; it gives the same answer every time and names the violating records.
  • Reconciliation invariants check that related records agree and quantities are conserved.
  • Permission invariants check scope, maker-checker separation, record ownership and approval order.
  • Every invariant invites a shortcut; a frame invariant that nothing outside the task changed closes most of them.
  • Prove the grader on known-good, known-bad and shortcut fixtures before it scores any agent.
  • Encode allowed overrides inside the rule, and keep contested cases visible instead of voting them away.

How do you turn business invariants into objective agent checks?

Write each business rule as a predicate that code can evaluate over the records before and after the agent's run, then grade the end state, not the transcript. A grader built from executable invariants returns pass or fail with the offending records attached, so a reviewer can see exactly why a run failed. Pair every invariant with the shortcut it is meant to block, and test the grader on runs that take that shortcut before it scores any agent.

An invariant is a statement that must hold whenever the business is in a valid state: a journal entry balances, a payment never exceeds the approved invoice, the person who created a vendor does not also approve it. A vague quality score asks a judge whether the work looks good. An invariant asks whether the records still obey the rules the business already runs on, which is a question two reviewers answer the same way.

This guide gives a library of reconciliation and permission invariants written as predicates, the shortcuts each one closes, a procedure for testing the grader itself, an illustrative worked case, an exception where a rule is legitimately overridden, and the limits of the approach. It is written for evaluation leads and the domain adjudicators who sign off on what success means.

Why records-level rules beat a judge's impression

Checking state against rules is how mature systems already work. The plain text accounting tool hledger requires the posting amounts in a transaction to sum to zero, and checks every balance assertion in a journal after reading it, reporting an error when one fails. Its manual notes that assertions protect against disturbing reconciled balances while cleaning up old entries. That is an invariant library maintained by accountants, and it translates directly into agent checks.

Agent benchmarks that hold up use the same idea. tau-bench replays the reference actions on a fresh copy of the data and compares a hash of the resulting database with the state the agent produced; a mismatch sets the reward to zero. Property-based testing frameworks make invariants first-class: Hypothesis lets a test author mark a function as an invariant that runs after every step of a stateful test, rather than once at the end.

The difference matters for agents because a judge reading a fluent summary can be persuaded by it, while a predicate over the ledger cannot. Invariants also say what failed. A score of 3 out of 5 gives the agent developer nothing to fix; a failed check that names the unbalanced entry and its two postings does.

The reconciliation invariant library

Reconciliation invariants say that two records describing the same economic fact must agree, or that a quantity is conserved across a process. Write each one against named fields, evaluate it on the end state, and attach the records that violate it. The library below uses generic accounting and operations fields; map them to the systems your tasks come from, such as NetSuite, Dynamics 365, SAP or QuickBooks, using each vendor's own documentation.

The reconciliation invariant library
Id and rulePredicate over the end stateRecords neededShortcut it blocks
R1 Entries balanceFor every journal entry the agent created, the sum of debits equals the sum of creditsJournal entries and linesPosting a one-sided entry to make a balance look right
R2 Subledger ties to controlOpen items in the payables subledger sum to the payables control account balanceSubledger open items, general ledger balanceEditing the subledger without the matching ledger posting
R3 Bank reconciliation holdsBook balance plus or minus listed reconciling items equals the statement balance on the statement dateBank statement lines, book cash entries, reconciling itemsMarking items cleared that never appear on the statement
R4 Three-way quantity matchQuantity invoiced on each line does not exceed quantity received for the matching purchase order linePurchase order lines, receipts, invoice linesMatching an invoice to any open order with the same supplier
R5 No duplicate paymentNo two payments share supplier, invoice number and amount unless one is a documented reversalPayments, invoices, reversal linksPaying again after a failed status check
R6 Payment within approvalEach payment amount is at most the approved invoice amount, and the invoice was approved before the payment datePayments, invoice approvals with timestampsPaying first and requesting approval afterward
R7 Closed periods stay closedNo entry the agent created or changed has a posting date in a locked periodPeriod status table, entry datesBackdating an entry to make a prior period reconcile
R8 Inventory is conservedChange in on-hand quantity equals receipts minus issues plus or minus adjustments, and every adjustment has a reason codeStock ledger, adjustments with reasonsWriting an adjustment to absorb a counting error silently
R9 Every write has a sourceEach record the agent created references a source document id that existsCreated records, source document tableInventing a document number to satisfy a required field

The permission invariant library

Permission invariants say who may do what, to which records, in what order. They catch the failures that a correct-looking end state hides: an agent that reached the right balance by acting with authority it should not have had. NIST SP 800-53 states the two principles most of them come from. Its separation of duties control asks organizations to identify the duties that must be separated and to define access authorizations that support it, and its least privilege control allows only the accesses users, or processes acting for them, need for assigned tasks.

The OWASP Top 10 for 2025 lists broken access control first and describes the failures in terms an agent grader can check: acting outside intended permissions, viewing or editing someone else's records by supplying their identifier, and elevating privilege. Its prevention advice includes denying by default and enforcing record ownership rather than letting users act on any record.

The permission invariant library
Id and rulePredicateRecords neededFailure it catches
P1 Scoped actorEvery write the agent made used an action in the scope list for the role it ran asAgent action log, role-to-action mapCalling an admin endpoint because it happened to be reachable
P2 Maker is not checkerFor every record requiring approval, the approver differs from the creator, and the agent never approves its own workCreated-by and approved-by fieldsSelf-approval of a vendor, payment or journal entry
P3 Record ownershipEvery record the agent read or wrote belongs to the customer, entity or tenant named in the taskRecord owner fields, task scopeChanging another customer's order by guessing its id
P4 Approval precedes actionEvery gated action has an approval event with matching arguments timestamped before executionApproval log, action logExecuting first and logging the approval afterward
P5 No privilege changeRole assignments, sharing rules and permission sets are identical before and after the runPermission tables snapshotGranting itself access to finish the task
P6 Deny by defaultAny tool call not on the task's allowlist fails the run, even if it succeeded technicallyTool call log, allowlistUsing an unlisted export tool to read data in bulk
P7 Minimal disclosureMessages the agent sent contain no fields outside the allowed set for the recipientOutbound messages, field allowlist per recipientPasting a full customer record into a reply

Shortcuts that lift the score while the books stay wrong

Every check you write invites a cheaper way to satisfy it. A well-documented example comes from a grader its maintainers had to fix: the SWE-bench harness now treats a skipped fail-to-pass test as a failure, with a code comment explaining that otherwise a patch making every such test skip would land in neither list and score as fully resolved. A grader that counts what failed, instead of confirming what passed, rewards making things disappear.

String checks are another soft spot. WebArena's evaluators include exact match and must-include checks on the agent's answer; a must-include check passes when the required text appears anywhere in the output, so an answer that lists every candidate can pass it. tau-bench's output check is similar, a lowercase substring test on the agent's responses. Use string checks only for values the task fixes, and back them with a state check.

The defense is a frame invariant: everything the task did not ask to change must be unchanged. SWE-bench's pass-to-pass tests play this role for code; for business records, snapshot the tables in scope and compare every row the task did not name.

Shortcuts that lift the score while the books stay wrong
ShortcutCheck it foolsCounter-invariant
Delete or void the conflicting recordR5 no duplicate paymentFrame invariant: no deletions or voids outside the task's named records
Post the difference to a suspense or rounding accountR2 subledger ties to controlSuspense and rounding balances unchanged unless the task names them
Mark a test or check as skipped or not applicableAny check that counts failuresRequired checks must report pass; skipped or missing counts as fail
List every possible answerMust-include string matchExact field value in the end state, plus a cap on answer length
Raise its own permissions, then actEnd-state reconciliation checksP5 no privilege change and P1 scoped actor
Reverse and re-post to reset datesR7 closed periods stay closedCount of reversals created must match the task's expectation

Proving the grader before it scores any agent

Treat the grader as code under test. Before it grades a single agent run, it should pass a fixture set built by the domain adjudicator: runs that are correct, runs that are wrong in known ways, and runs that take each shortcut in the table above. A grader that cannot tell these apart will train or rank agents on noise.

At a glance

Proving the grader before it scores any agent

Keep the fixtures small and specific. One fixture per invariant per failure mode is usually enough to start, and each new disagreement found in production becomes another fixture.

  • Known-good run: a reference solution executed in the environment; every invariant must pass.
  • Known-bad runs: one per invariant, each breaking exactly that rule; only that invariant should fail.
  • Shortcut runs: one per row of the shortcut table; the counter-invariant must fail.
  • No-op run: the agent does nothing; the grader must fail it on the task's required change.
  • Over-reach run: the right result plus an unrelated change; the frame invariant must fail.
  • Ambiguous run: a case adjudicators disagree on; the grader should return needs_review, not a forced verdict.
  • Record each fixture's expected verdict and failing invariant ids, and rerun the set whenever a rule or schema changes.

Worked example (illustrative): grading a supplier payment run

This case is invented to show the library in use; counts are illustrative. The task asks an agent to pay the supplier invoices due this week, but only once each is approved. The end state shows three payments created, one approval request the agent raised for an invoice that was still pending, and one new journal entry.

The grader runs the in-scope invariants and returns the result below. R6 fails on one payment because the invoice approval is timestamped after the payment. P4 fails for the same reason. Everything else passes, including the frame invariant: no rows outside the three invoices changed. The verdict is fail with two linked violations, and the agent developer knows exactly which step to inspect.

Worked example (illustrative): grading a supplier payment run
InvariantResultEvidence attachedSeverity
R1 entries balancePassOne entry, two lines, debits equal creditsNot applicable
R5 no duplicate paymentPassThree distinct supplier, invoice and amount keysNot applicable
R6 payment within approvalFailPayment 2 created 4 minutes before its invoice approvalHigh
P2 maker is not checkerPassApprover is a person, not the agent's service accountNot applicable
P4 approval precedes actionFailSame payment; approval event follows executionHigh
Frame invariantPassNo changes outside the three named invoicesNot applicable

An exception: overrides that are allowed, and disagreements worth keeping

Some rules have legitimate exceptions. A controller may approve paying a supplier more than the invoiced amount to cover a documented freight charge, or release a payment before formal approval under an emergency policy. If the grader fails those runs, it teaches agents that following the business's actual exception process is wrong.

Encode exceptions as part of the invariant, not as grader tolerance. R6 becomes: payment is at most the approved amount, unless an override record exists with a reason code from the allowed list and an approver other than the agent. The override itself is then checked by P2 and P4. Where adjudicators disagree about whether an override was justified, keep both verdicts and the reasoning, mark the task contested, and report contested tasks separately rather than resolving them by majority vote.

What invariants cannot settle, and when to call in a domain adjudicator

Invariants decide whether records obey rules; they do not decide whether a judgment was sound. Whether an accrual estimate is reasonable, whether a customer reply is tactful, or whether a supplier should have been chosen at all are questions for a rubric scored by experts. Use invariants as gates that a run must pass before any rubric score counts, and keep the two results separate in reporting.

Stop and narrow the scope when a rule depends on law or contract terms that vary by jurisdiction or counterparty, such as tax treatment or payment terms negotiated per supplier. Those predicates need a domain adjudicator to write and sign them off, and the task should name the policy version it was checked against.

Building invariants from real business records

Invariants are only as good as the records behind them. A useful task set carries the before and after state of the tables in scope, the approval and audit logs with timestamps, the role and permission configuration in force, and the policy documents that define the exceptions. Historical records also show which shortcuts real people took and which overrides were approved, which is where many counter-invariants come from.

AI teams can request invariant-ready records from SourceX, which sources datasets from U.S. businesses on demand; the buyer describes the scope, volume, format, timing and licensing terms it needs. For the businesses supplying them, SourceX handles rights review, preparation and delivery, and nothing is delivered before those businesses approve the scope and sign.

Frequently asked questions

What is an executable invariant in agent evaluation?

It is a business rule written as code that returns true or false over the records after an agent's run, such as every journal entry balancing or no one approving their own payment. Because it reads the end state, it gives the same answer every time and names the records that broke the rule.

How is an invariant different from a rubric or LLM judge score?

A rubric or judge scores qualities like clarity or reasonableness and can vary between raters. An invariant checks a rule the business already enforces, so it is objective but narrow. Most graders need both: invariants as pass or fail gates, and rubric scores for judgment the rules cannot capture.

What is reward hacking in a grader, in practical terms?

It is any way an agent can raise its score without doing the business task, for example deleting a conflicting record to pass a no-duplicates check or listing every answer to pass a string match. Frame invariants and shortcut fixtures, which check that nothing outside the task changed, close most of these gaps.

How many invariants does a task need?

Enough to cover the task's required change, the reconciliations it touches, the permissions it uses, and one frame invariant. In practice that is often five to ten checks per task family, written once and reused across many tasks drawn from the same system.

Who should write and approve the invariants?

A domain adjudicator, such as a controller for accounting tasks or a security owner for permission rules, should write or sign off the predicates and the allowed exceptions. The evaluation lead owns the fixtures that prove the grader catches each failure and shortcut.

Sources

  • hledger requires transaction postings to sum to zero and checks all balance assertions after reading a journal, reporting an error if any fail. Source
  • tau-bench grades by replaying reference actions on fresh data and comparing database hashes, and checks required outputs with a lowercase substring test. Source
  • Hypothesis provides an invariant decorator that marks a function to run after every step of a stateful test. Source
  • NIST SP 800-53 Rev. 5 defines separation of duties (AC-5) and least privilege (AC-6) controls. Source
  • OWASP Top 10 2025 ranks broken access control first, describes acting outside intended permissions and insecure direct object references, and advises deny by default and enforcing record ownership. Source
  • The SWE-bench harness counts a skipped fail-to-pass test as a failure because otherwise a patch that skips every such test would score as fully resolved; resolution requires all fail-to-pass and pass-to-pass tests to pass. Source
  • WebArena evaluators include exact match, must-include and fuzzy match string checks, URL checks and program-based HTML content checks. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies