AI uses for records
Counting the cleanup: an AI reconciliation review effort dataset
By SourceX Editorial · Reviewed by Noah Loul ·
8 sections
Short answer
An AI reconciliation review effort dataset should count, item by item, what a person had to touch, override, reopen or leave open after the matcher ran, plus accepted items that a blind re-review later found wrong. Accuracy alone hides that work. In one illustrative batch of 12 receipts, final pairings were 80 percent correct, yet people touched two thirds of them and one in six stayed open.
Agent run
Counting the cleanup
Key takeaways
- Count review effort per item, never per click, against fixed denominators, and publish all seven counts together.
- An unchanged AI match is not a reviewed match; automation-bias research ties acceptance to workload and time pressure, so a blind re-review sample must measure escaped errors.
- In Midday's open-source matcher, automatic attachments are stored with the same confirmed status a person's confirmation writes, and the calibration accuracy counts both.
- The same code keeps one status per receipt and transaction pair, logs no event for a decline or unmatch, and records no actor for a manual match, so final states cannot rebuild review effort.
- Pair every effort count with a qualified reviewer's treatment verdict, because matching amounts do not show the accounting was right.
What should an AI reconciliation review effort dataset count?
It should count, for every item the AI processed, what a person had to touch, override, reopen or leave open, plus the accepted items a blind re-review later found wrong, each against a stated denominator and alongside active minutes. Accuracy sits beside those counts as one column; it cannot replace them, because two matchers with the same accuracy can leave very different amounts of work behind.
Finance teams buying or building a reconciliation agent want to know how much human work remains after it runs, and that question has no answer in a match rate. The unit is the item, such as a receipt, a bank line or an open invoice, never the click, so a group confirmation that settles three items counts as one action touching three items.
The method below has four parts: an audit of what an open-source receipt matcher's code records when people review its output, seven counting rules shown on one illustrative batch, a casebook of eight exceptions and a rubric for packets. No time-saving or accuracy figure from any real deployment is reported, because none was measured for this page.
Why a match rate cannot stand in for review effort
Because it describes the machine's output, not the person's work, and because an accepted match may reflect the reviewer's workload as much as the match's quality. The same match rate can come with one reviewer glance per item or with a long tail of manual searches, reopened periods and items still open at the cutoff.
Research on automation bias in clinical decision support explains why acceptance is weak evidence. A 2012 systematic review in the Journal of the American Medical Informatics Association by Goddard, Roudsari and Wyatt screened 13,821 papers and included 74. It found that workload, task complexity and time constraints mediate how readily people follow automated advice, and it listed design mitigators such as attaching updated confidence levels to output and offering information rather than recommendations. A matched item nobody changed may simply be an item nobody had time to check.
Auditors reached the same conclusion about review controls. The PCAOB's Staff Audit Practice Alert No. 11 (October 24, 2013) describes reviews that compare recorded amounts to expected amounts and investigate significant differences, and it says that verifying a review was signed off provides little or no evidence by itself about the control's effectiveness. PCAOB staff guidance treats the criteria for investigation as the threshold for follow-up. A review-effort dataset has to record what happened to each flagged item, not that a sign-off exists.
What one open-source matcher records about its reviewers
Less than a review-effort count needs: a confirmation leaves a trace, while declines, reopenings and manual matches leave only a final status or nothing at all. The evidence is the source code of Midday, an AGPL-licensed business tool that its README describes as built for freelancers, contractors, consultants and solo entrepreneurs, with an inbox that matches incoming receipts and invoices to bank transactions. Its documentation says high-confidence receipt matches are attached automatically, uncertain ones are flagged for review and the matcher learns from confirmed, declined and unmatched feedback.
The audit read the code at commit 5158731 of June 13, 2026, with a script that parses the code and never runs it. Automatically attached suggestions are written with status confirmed at the moment they are created, the same status a person's confirmation writes; only the match type and an empty user field tell them apart. The suggestion table holds one row per receipt and transaction pair, so when a person unmatches an earlier match, the row's status changes to unmatched and the new user and time replace those of the confirmation.
The activity log's 32 event types include one for a human confirmation, inbox_match_confirmed, and machine events such as inbox_auto_matched, but none for a decline or an unmatch. The manual-match procedure passes no user identifier, and the function it calls receives only the item, the transaction and the team. Deleting a receipt deletes its suggestion rows. Team calibration reads 90 days of confirmed, declined and unmatched suggestions without filtering on match type and computes a field named suggestedMatchAccuracy as confirmed rows over all of them, so automatic attachments count toward the confirmed share.
None of this is a defect in a bookkeeping product built to match receipts rather than to study its users. It shows why a buyer cannot rebuild review effort from a system's final state: the packet needs an event log written at the moment each action happens. The table summarizes what the code writes for each action:
| Action | What the code writes | Actor recorded | What survives later steps |
|---|---|---|---|
| Automatic attachment above the threshold | Suggestion row with status confirmed and match type auto_matched; an inbox_auto_matched notification | No user | Counted with human confirmations in team calibration |
| Person confirms a suggestion | Status confirmed, an inbox_match_confirmed activity, and pending suggestions for grouped receipts set to confirmed | User and time | Status replaced if later unmatched; the activity stays |
| Person declines a suggestion | Status declined; the receipt returns to pending | User and time | Deleted with the receipt; no activity type exists |
| Person unmatches an earlier match | Status unmatched on the confirmed row | User and time, overwriting the confirmation's | The confirmation's actor and time are gone |
| Person matches by hand | Receipt linked to the transaction and an attachment created | None passed | No suggestion row and no activity |
| Person deletes the receipt | Every suggestion row for it deleted | None | Nothing |
Seven counts, their denominators and one illustrative batch
Count per item with fixed denominators and publish all seven counts together, because each one alone flatters the system or the reviewer. The table applies the rules to an illustrative batch of 12 receipts built for this page: five attached automatically, five suggested for confirmation and two with no proposal.
In the batch, one automatic match was unmatched after the period review and matched by hand; one click confirmed two grouped suggestions; two suggestions were declined, one rematched by hand and one left open; one receipt with no proposal was matched by hand and the other stayed open; one correct pairing needed a tax-field fix. A blind re-review then found two accepted pairings wrong: a duplicate receipt attached automatically and a high-confidence suggestion a person confirmed to the wrong vendor.
Read together, the counts tell a different story from any one of them. Final pairings were 80 percent correct and 70 percent of the AI's pairings were accepted, yet people touched 8 of the 12 receipts with 9 actions, reopened one and left 2 open. A status table like the one audited above would show 7 of 10 decided suggestions as confirmed, though people confirmed only 3 of those 7. Active minutes per touched item belong beside these counts, taken from tool events rather than recollection; the batch has no time column because inventing one would defeat its purpose. The counting rules and the batch's values:
| Count | Numerator | Denominator | Illustrative batch |
|---|---|---|---|
| Accuracy | Final pairings the blind re-review confirms | Items with a final pairing | 8 of 10 (80.0%) |
| Accepted | AI pairings never declined or unmatched | Items with an AI pairing | 7 of 10 (70.0%) |
| Touched | Items with at least one action by a person, group actions included | All items | 8 of 12 (66.7%) |
| Overridden | AI pairings a person declined or unmatched | Items with an AI pairing | 3 of 10 (30.0%) |
| Reopened | Items paired, then unpaired after pairing | A count, each with its date | 1 |
| Open at cutoff | Items with no pairing at the period cutoff | All items | 2 of 12 (16.7%) |
| Escaped | Accepted items the blind re-review found wrong | Accepted items re-reviewed | 2 of 7 (28.6%) |
Casebook: eight reconciliation exceptions and the review each one costs
The eight cases are illustrative and drawn from common receipt-to-bank exceptions, not from any company's records. Each one names the counts it adds, which is how a casebook turns into a scoring key; a qualified accountant should confirm the expected treatment before real packets are labeled.
| Case (illustrative) | What the AI did | What the person had to do | Counts it adds |
|---|---|---|---|
| One card charge, two receipts | Matched one receipt and left the other pending | Group both receipts under the charge | Touched; open until grouped |
| The same receipt forwarded twice | Attached the copy to the already-matched charge | Nothing at the time; the re-review caught it | Escaped |
| Foreign-currency charge differs from the receipt total | Suggested at high confidence | Check the rate and fee line, then confirm | Touched |
| Refund posted as a new bank line | Matched the refund receipt to the original purchase | Unmatch, then match the refund line by hand | Overridden; touched |
| Merchant renamed by the card processor | No proposal; the name score was too low | Search and match by hand | Touched |
| Error found after the month was reviewed | Attached automatically in the reviewed month | Reopen, correct and note the period | Reopened; overridden |
| Personal purchase on a company card | Suggested a business charge | Decline and record why the receipt stays unmatched | Overridden; open with a reason |
| Tax amount misread from the image | Pairing right, tax field wrong | Correct the tax field | Touched; pairing accepted |
Reviewer rubric: six tests for a review-effort label
A packet earns a complete review-effort label only when all six tests below hold. Two reviewers apply them independently, an accountant with authority for the entity settles disagreements, and the packet records who decided.
At a glance
Reviewer rubric: six tests for a review-effort label
- Disposition: every item ends the period paired, open with a reason or excluded with a reason.
- Event trail: every action carries an actor, a time, and the state before and after, written when it happened.
- Machine versus person: every count can be split into automatic and human decisions.
- Reopenings: each one names the reviewed period or sign-off it reopened.
- Blind re-review: a random sample of accepted items, drawn with a published seed and sample size, is re-checked by someone who cannot see the AI's proposal or score.
- Time: active minutes per touched item come from tool events, with idle gaps excluded by a stated rule.
Is a confirmed match a reviewed match?
No. A confirmation proves that someone clicked, and matching amounts and dates prove arithmetic agreement; neither shows the accounting treatment was right or that the person had authority to accept it. In the casebook, the personal purchase matches a business charge on amount and date, and the correct review outcome is a decline.
The strongest alternative view is that time alone measures effort, so the counts are redundant. Time without the counts cannot separate a careful reviewer from a fast one, or a quick correct confirmation from a quick wrong one; only the escaped count exposes the second. The reverse objection, that high accuracy makes effort irrelevant, fails on the same batch: accuracy was 80 percent while one in six receipts was still open.
So the label pairs each effort count with a treatment verdict from a qualified reviewer. Arithmetic correctness is the entry ticket, and the review-effort columns record the price of reaching a defensible reconciliation.
Requesting review-effort packets, and the limits of this design
AI teams can ask SourceX for reconciliation packets of this kind; it locates records for buyers against a written request instead of selling from a catalog. A request that lists the fields below, the systems that must supply the event history and the acceptance test the packets must pass lets a supplier judge quickly whether its records can carry the label. Nothing here implies records already on hand, an accepted delivery or a gain in any model.
At a glance
Requesting review-effort packets, and the limits of this design
Limits of this design: the code audit covers one open-source product at one commit, and a hosted version may log more than its public code shows. The batch is illustrative, so its percentages describe the counting rules, not any real matcher. Escaped-error rates need a blind sample large enough for a useful confidence interval, which depends on the error rate itself. The next evidence needed is an authorized, de-identified event log from a live close, labeled by two reviewers. Each packet carries these fields:
- Item identifier, type and period, with the cutoff date.
- The AI's proposal, its scores, its match type and the model or rule version.
- Every event: actor type (system or person), role, action, state before and after, timestamp.
- Final disposition and the reason for any open or excluded item.
- Blind re-review verdict, reviewer qualification and sampling seed.
- Active minutes per touched item and the idle-gap rule used.
Frequently asked questions
What is review effort after an AI reconciliation?
It is the work people still do after the matcher runs: items they touch, AI pairings they decline or unmatch, closed periods they reopen, items left open at the cutoff and accepted items later found wrong, plus the active minutes behind them. It is counted per item against a fixed denominator.
Why is the acceptance rate of AI matches a weak measure?
An unchanged match may never have been checked. Research on automation bias links acceptance of automated advice to workload, task complexity and time pressure, so acceptance needs a blind re-review sample before it says anything about correctness.
How should a reopened reconciliation be counted?
As its own event on the item, with the date and the period or sign-off it reopened, and as an override when the reopening changed the AI's pairing. A system that overwrites the earlier status cannot supply this count, so the event must be logged when it happens.
Can review effort be measured from a system's audit log?
Only if the log keeps every action with its actor and before-and-after state. In the open-source matcher audited here, declines and unmatches change a single status field and leave no activity event, and a manual match records no actor.
Does a confirmed match prove the reconciliation is right?
No. It proves a click and arithmetic agreement. Treatment, such as whether a purchase is personal or capital, and the reviewer's authority need a qualified reviewer's verdict alongside the effort counts.
Sources
- Midday's README describes an all-in-one tool for freelancers, contractors, consultants and solo entrepreneurs whose Magic Inbox matches incoming invoices or receipts to transactions. Source
- Midday's receipt-matching documentation says high-confidence matches are attached automatically, uncertain matches are flagged for review and matching learns from confirmed, declined and unmatched feedback. Source
- Midday's database schema defines the match-suggestion table (one row per receipt and transaction pair, a single status field, match types) and the activity log's event types. Source
- Midday's matching queries create automatic matches with status confirmed, record a user and an activity on confirmation, and record a user but no activity on a decline. Source
- Midday's inbox queries set an earlier confirmed suggestion to unmatched with a new user and time, match by hand without an actor, and delete suggestion rows when a receipt is deleted. Source
- Midday's team calibration reads 90 days of confirmed, declined and unmatched suggestions without a match-type filter and computes accuracy as confirmed over all. Source
- Midday's inbox API router passes the signed-in user to confirm, decline and unmatch procedures but not to the manual match procedure. Source
- Goddard, Roudsari and Wyatt (JAMIA, 2012) reviewed automation bias: 74 of 13,821 papers met inclusion criteria; workload, task complexity and time constraints were mediators, and confidence levels on output were among the mitigators. Source
- PCAOB Staff Audit Practice Alert No. 11 (October 24, 2013) says verifying that a review was signed off provides little or no evidence by itself about the control's effectiveness. Source
- PCAOB staff guidance on auditing internal control describes criteria for investigation as the threshold for follow-up in detective controls. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.