Skip to content

AI uses for records

Counting the cleanup: an AI reconciliation review effort dataset

By SourceX Editorial · Reviewed by Noah Loul ·

8 sections

Short answer

An AI reconciliation review effort dataset should count, item by item, what a person had to touch, override, reopen or leave open after the matcher ran, plus accepted items that a blind re-review later found wrong. Accuracy alone hides that work. In one illustrative batch of 12 receipts, final pairings were 80 percent correct, yet people touched two thirds of them and one in six stayed open.

Agent run

Counting the cleanup

This page covers counting the cleanup: an AI reconciliation review effort dataset.Illustrative

Key takeaways

  • Count review effort per item, never per click, against fixed denominators, and publish all seven counts together.
  • An unchanged AI match is not a reviewed match; automation-bias research ties acceptance to workload and time pressure, so a blind re-review sample must measure escaped errors.
  • In Midday's open-source matcher, automatic attachments are stored with the same confirmed status a person's confirmation writes, and the calibration accuracy counts both.
  • The same code keeps one status per receipt and transaction pair, logs no event for a decline or unmatch, and records no actor for a manual match, so final states cannot rebuild review effort.
  • Pair every effort count with a qualified reviewer's treatment verdict, because matching amounts do not show the accounting was right.

What should an AI reconciliation review effort dataset count?

It should count, for every item the AI processed, what a person had to touch, override, reopen or leave open, plus the accepted items a blind re-review later found wrong, each against a stated denominator and alongside active minutes. Accuracy sits beside those counts as one column; it cannot replace them, because two matchers with the same accuracy can leave very different amounts of work behind.

Finance teams buying or building a reconciliation agent want to know how much human work remains after it runs, and that question has no answer in a match rate. The unit is the item, such as a receipt, a bank line or an open invoice, never the click, so a group confirmation that settles three items counts as one action touching three items.

The method below has four parts: an audit of what an open-source receipt matcher's code records when people review its output, seven counting rules shown on one illustrative batch, a casebook of eight exceptions and a rubric for packets. No time-saving or accuracy figure from any real deployment is reported, because none was measured for this page.

Why a match rate cannot stand in for review effort

Because it describes the machine's output, not the person's work, and because an accepted match may reflect the reviewer's workload as much as the match's quality. The same match rate can come with one reviewer glance per item or with a long tail of manual searches, reopened periods and items still open at the cutoff.

Research on automation bias in clinical decision support explains why acceptance is weak evidence. A 2012 systematic review in the Journal of the American Medical Informatics Association by Goddard, Roudsari and Wyatt screened 13,821 papers and included 74. It found that workload, task complexity and time constraints mediate how readily people follow automated advice, and it listed design mitigators such as attaching updated confidence levels to output and offering information rather than recommendations. A matched item nobody changed may simply be an item nobody had time to check.

Auditors reached the same conclusion about review controls. The PCAOB's Staff Audit Practice Alert No. 11 (October 24, 2013) describes reviews that compare recorded amounts to expected amounts and investigate significant differences, and it says that verifying a review was signed off provides little or no evidence by itself about the control's effectiveness. PCAOB staff guidance treats the criteria for investigation as the threshold for follow-up. A review-effort dataset has to record what happened to each flagged item, not that a sign-off exists.

What one open-source matcher records about its reviewers

Less than a review-effort count needs: a confirmation leaves a trace, while declines, reopenings and manual matches leave only a final status or nothing at all. The evidence is the source code of Midday, an AGPL-licensed business tool that its README describes as built for freelancers, contractors, consultants and solo entrepreneurs, with an inbox that matches incoming receipts and invoices to bank transactions. Its documentation says high-confidence receipt matches are attached automatically, uncertain ones are flagged for review and the matcher learns from confirmed, declined and unmatched feedback.

The audit read the code at commit 5158731 of June 13, 2026, with a script that parses the code and never runs it. Automatically attached suggestions are written with status confirmed at the moment they are created, the same status a person's confirmation writes; only the match type and an empty user field tell them apart. The suggestion table holds one row per receipt and transaction pair, so when a person unmatches an earlier match, the row's status changes to unmatched and the new user and time replace those of the confirmation.

The activity log's 32 event types include one for a human confirmation, inbox_match_confirmed, and machine events such as inbox_auto_matched, but none for a decline or an unmatch. The manual-match procedure passes no user identifier, and the function it calls receives only the item, the transaction and the team. Deleting a receipt deletes its suggestion rows. Team calibration reads 90 days of confirmed, declined and unmatched suggestions without filtering on match type and computes a field named suggestedMatchAccuracy as confirmed rows over all of them, so automatic attachments count toward the confirmed share.

None of this is a defect in a bookkeeping product built to match receipts rather than to study its users. It shows why a buyer cannot rebuild review effort from a system's final state: the packet needs an event log written at the moment each action happens. The table summarizes what the code writes for each action:

What one open-source matcher records about its reviewers
ActionWhat the code writesActor recordedWhat survives later steps
Automatic attachment above the thresholdSuggestion row with status confirmed and match type auto_matched; an inbox_auto_matched notificationNo userCounted with human confirmations in team calibration
Person confirms a suggestionStatus confirmed, an inbox_match_confirmed activity, and pending suggestions for grouped receipts set to confirmedUser and timeStatus replaced if later unmatched; the activity stays
Person declines a suggestionStatus declined; the receipt returns to pendingUser and timeDeleted with the receipt; no activity type exists
Person unmatches an earlier matchStatus unmatched on the confirmed rowUser and time, overwriting the confirmation'sThe confirmation's actor and time are gone
Person matches by handReceipt linked to the transaction and an attachment createdNone passedNo suggestion row and no activity
Person deletes the receiptEvery suggestion row for it deletedNoneNothing
Two panels for one illustrative receipt: on the left, four events in order (suggested to a reviewer, confirmed, unmatched, matched by hand); on the right, the single suggestion row that remains, showing match type high_confidence and status unmatched, having lost who confirmed it and when, with no record of who matched it by hand.
What one receipt's history leaves in Midday's match-suggestion row, from this article's code audit (commit 5158731); the confirmation survives only in the activity log. The receipt is illustrative.

Seven counts, their denominators and one illustrative batch

Count per item with fixed denominators and publish all seven counts together, because each one alone flatters the system or the reviewer. The table applies the rules to an illustrative batch of 12 receipts built for this page: five attached automatically, five suggested for confirmation and two with no proposal.

In the batch, one automatic match was unmatched after the period review and matched by hand; one click confirmed two grouped suggestions; two suggestions were declined, one rematched by hand and one left open; one receipt with no proposal was matched by hand and the other stayed open; one correct pairing needed a tax-field fix. A blind re-review then found two accepted pairings wrong: a duplicate receipt attached automatically and a high-confidence suggestion a person confirmed to the wrong vendor.

Read together, the counts tell a different story from any one of them. Final pairings were 80 percent correct and 70 percent of the AI's pairings were accepted, yet people touched 8 of the 12 receipts with 9 actions, reopened one and left 2 open. A status table like the one audited above would show 7 of 10 decided suggestions as confirmed, though people confirmed only 3 of those 7. Active minutes per touched item belong beside these counts, taken from tool events rather than recollection; the batch has no time column because inventing one would defeat its purpose. The counting rules and the batch's values:

Seven counts, their denominators and one illustrative batch
CountNumeratorDenominatorIllustrative batch
AccuracyFinal pairings the blind re-review confirmsItems with a final pairing8 of 10 (80.0%)
AcceptedAI pairings never declined or unmatchedItems with an AI pairing7 of 10 (70.0%)
TouchedItems with at least one action by a person, group actions includedAll items8 of 12 (66.7%)
OverriddenAI pairings a person declined or unmatchedItems with an AI pairing3 of 10 (30.0%)
ReopenedItems paired, then unpaired after pairingA count, each with its date1
Open at cutoffItems with no pairing at the period cutoffAll items2 of 12 (16.7%)
EscapedAccepted items the blind re-review found wrongAccepted items re-reviewed2 of 7 (28.6%)

Casebook: eight reconciliation exceptions and the review each one costs

The eight cases are illustrative and drawn from common receipt-to-bank exceptions, not from any company's records. Each one names the counts it adds, which is how a casebook turns into a scoring key; a qualified accountant should confirm the expected treatment before real packets are labeled.

Casebook: eight reconciliation exceptions and the review each one costs
Case (illustrative)What the AI didWhat the person had to doCounts it adds
One card charge, two receiptsMatched one receipt and left the other pendingGroup both receipts under the chargeTouched; open until grouped
The same receipt forwarded twiceAttached the copy to the already-matched chargeNothing at the time; the re-review caught itEscaped
Foreign-currency charge differs from the receipt totalSuggested at high confidenceCheck the rate and fee line, then confirmTouched
Refund posted as a new bank lineMatched the refund receipt to the original purchaseUnmatch, then match the refund line by handOverridden; touched
Merchant renamed by the card processorNo proposal; the name score was too lowSearch and match by handTouched
Error found after the month was reviewedAttached automatically in the reviewed monthReopen, correct and note the periodReopened; overridden
Personal purchase on a company cardSuggested a business chargeDecline and record why the receipt stays unmatchedOverridden; open with a reason
Tax amount misread from the imagePairing right, tax field wrongCorrect the tax fieldTouched; pairing accepted

Reviewer rubric: six tests for a review-effort label

A packet earns a complete review-effort label only when all six tests below hold. Two reviewers apply them independently, an accountant with authority for the entity settles disagreements, and the packet records who decided.

At a glance

Reviewer rubric: six tests for a review-effort label

  • Disposition: every item ends the period paired, open with a reason or excluded with a reason.
  • Event trail: every action carries an actor, a time, and the state before and after, written when it happened.
  • Machine versus person: every count can be split into automatic and human decisions.
  • Reopenings: each one names the reviewed period or sign-off it reopened.
  • Blind re-review: a random sample of accepted items, drawn with a published seed and sample size, is re-checked by someone who cannot see the AI's proposal or score.
  • Time: active minutes per touched item come from tool events, with idle gaps excluded by a stated rule.

Is a confirmed match a reviewed match?

No. A confirmation proves that someone clicked, and matching amounts and dates prove arithmetic agreement; neither shows the accounting treatment was right or that the person had authority to accept it. In the casebook, the personal purchase matches a business charge on amount and date, and the correct review outcome is a decline.

The strongest alternative view is that time alone measures effort, so the counts are redundant. Time without the counts cannot separate a careful reviewer from a fast one, or a quick correct confirmation from a quick wrong one; only the escaped count exposes the second. The reverse objection, that high accuracy makes effort irrelevant, fails on the same batch: accuracy was 80 percent while one in six receipts was still open.

So the label pairs each effort count with a treatment verdict from a qualified reviewer. Arithmetic correctness is the entry ticket, and the review-effort columns record the price of reaching a defensible reconciliation.

Requesting review-effort packets, and the limits of this design

AI teams can ask SourceX for reconciliation packets of this kind; it locates records for buyers against a written request instead of selling from a catalog. A request that lists the fields below, the systems that must supply the event history and the acceptance test the packets must pass lets a supplier judge quickly whether its records can carry the label. Nothing here implies records already on hand, an accepted delivery or a gain in any model.

At a glance

Requesting review-effort packets, and the limits of this design

Limits of this design: the code audit covers one open-source product at one commit, and a hosted version may log more than its public code shows. The batch is illustrative, so its percentages describe the counting rules, not any real matcher. Escaped-error rates need a blind sample large enough for a useful confidence interval, which depends on the error rate itself. The next evidence needed is an authorized, de-identified event log from a live close, labeled by two reviewers. Each packet carries these fields:

  • Item identifier, type and period, with the cutoff date.
  • The AI's proposal, its scores, its match type and the model or rule version.
  • Every event: actor type (system or person), role, action, state before and after, timestamp.
  • Final disposition and the reason for any open or excluded item.
  • Blind re-review verdict, reviewer qualification and sampling seed.
  • Active minutes per touched item and the idle-gap rule used.

Frequently asked questions

What is review effort after an AI reconciliation?

It is the work people still do after the matcher runs: items they touch, AI pairings they decline or unmatch, closed periods they reopen, items left open at the cutoff and accepted items later found wrong, plus the active minutes behind them. It is counted per item against a fixed denominator.

Why is the acceptance rate of AI matches a weak measure?

An unchanged match may never have been checked. Research on automation bias links acceptance of automated advice to workload, task complexity and time pressure, so acceptance needs a blind re-review sample before it says anything about correctness.

How should a reopened reconciliation be counted?

As its own event on the item, with the date and the period or sign-off it reopened, and as an override when the reopening changed the AI's pairing. A system that overwrites the earlier status cannot supply this count, so the event must be logged when it happens.

Can review effort be measured from a system's audit log?

Only if the log keeps every action with its actor and before-and-after state. In the open-source matcher audited here, declines and unmatches change a single status field and leave no activity event, and a manual match records no actor.

Does a confirmed match prove the reconciliation is right?

No. It proves a click and arithmetic agreement. Treatment, such as whether a purchase is personal or capital, and the reviewer's authority need a qualified reviewer's verdict alongside the effort counts.

Sources

  • Midday's README describes an all-in-one tool for freelancers, contractors, consultants and solo entrepreneurs whose Magic Inbox matches incoming invoices or receipts to transactions. Source
  • Midday's receipt-matching documentation says high-confidence matches are attached automatically, uncertain matches are flagged for review and matching learns from confirmed, declined and unmatched feedback. Source
  • Midday's database schema defines the match-suggestion table (one row per receipt and transaction pair, a single status field, match types) and the activity log's event types. Source
  • Midday's matching queries create automatic matches with status confirmed, record a user and an activity on confirmation, and record a user but no activity on a decline. Source
  • Midday's inbox queries set an earlier confirmed suggestion to unmatched with a new user and time, match by hand without an actor, and delete suggestion rows when a receipt is deleted. Source
  • Midday's team calibration reads 90 days of confirmed, declined and unmatched suggestions without a match-type filter and computes accuracy as confirmed over all. Source
  • Midday's inbox API router passes the signed-in user to confirm, decline and unmatch procedures but not to the manual match procedure. Source
  • Goddard, Roudsari and Wyatt (JAMIA, 2012) reviewed automation bias: 74 of 13,821 papers met inclusion criteria; workload, task complexity and time constraints were mediators, and confidence levels on output were among the mitigators. Source
  • PCAOB Staff Audit Practice Alert No. 11 (October 24, 2013) says verifying that a review was signed off provides little or no evidence by itself about the control's effectiveness. Source
  • PCAOB staff guidance on auditing internal control describes criteria for investigation as the threshold for follow-up in detective controls. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies