Skip to content

AI uses for records

Finance dataset error detection pilot: how to test correction history

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A finance dataset error detection pilot should compare matched records that differ in one way: one arm carries verified correction history, meaning the original entry, the error found, the fix and a reviewer's confirmation, while the other shows only the final state. Score both on a frozen, accountant-labeled test set, report missed errors and false alarms by error class, and publish the design before any result. No results are published here.

Record timeline

Finance dataset error

This page covers finance dataset error detection pilot: how to test correction history.Illustrative

Key takeaways

  • The pilot's question is narrow: does verified correction history raise detection of confirmed errors without raising false alarms on clean packets?
  • Matched arms that differ only in correction history keep company, period and volume from posing as the effect.
  • Of 180 in-force XBRL US Data Quality Committee rules, none mentions invoices, purchase orders or supporting documents; record-only checks stop where outside evidence starts.
  • Under illustrative assumptions, a 10-point recall gain needs 168 to 249 confirmed errors in the test set to detect with 80 percent power.
  • Report recall by error class, false alarms on clean look-alikes, both discordant counts and seed ranges, null results included.

What should a finance dataset error detection pilot decide?

Whether attaching verified correction history to finance records makes an AI system better at finding real errors, without raising false alarms, compared with matched records that show only the final state. That is the decision a finance dataset error detection pilot exists to settle: pay to collect, license or prepare correction history, or spend the same effort on more final-state volume.

The useful answer has a narrow shape. On a frozen test set of packets that qualified accountants have labeled, the arm trained or prompted with correction history should catch more confirmed errors in at least one error class, flag no more clean packets, and keep that gap across repeated training runs. A higher overall accuracy number alone would not settle it, because a detector that flags everything also finds every error.

This page publishes the design: definitions, the matched comparison, an exception casebook for the test set, a grading rubric, the number of errors the test needs and the reporting rules. It reports no results. Running the pilot needs authorized, de-identified records from companies that keep their corrections, and no such run has been published.

Outcome-linked, corrected, verified: the terms the test depends on

Four terms carry the design, and a pilot that leaves any of them loose will measure something else.

A record packet is one unit of finance work: an invoice, a journal entry, a payment or a reconciling item, together with its source documents, postings and approvals. An outcome-linked packet also carries what happened next, such as paid, disputed, reversed, adjusted or restated. Verified correction history is narrower still: the original entry, the error someone found, the correcting entry, who found it, the evidence they used and a qualified reviewer's confirmation that it was an error rather than a change of estimate or a judgment call.

The error definition should be as broad as auditing uses. In ISA 450, the international auditing standard on evaluating misstatements, a misstatement is a difference between the amount, classification, presentation or disclosure of a reported item and what the applicable financial reporting framework requires, and misstatements can arise from error or fraud. An entry can therefore balance, add up and still be wrong because it sits in the wrong account or the wrong period.

Detection is scored per packet: a flag counts only when it names an error class and points to the evidence for it. That rule keeps a detector from earning credit for a vague warning that happens to land on a bad packet.

What 180 public filing rules can check, and what they leave to evidence

Mostly relationships inside the document. The XBRL US Data Quality Committee publishes its rules for checking U.S. GAAP filings, with a documentation file per rule, on GitHub. At commit 5f8cd32 (October 23, 2026) the repository holds 185 U.S. rule documents, 180 of them in force once rules marked removed after public review or retired are set aside. Classified by title, 43 test arithmetic or balance relationships, 77 test which element, axis or member a value was tagged with, 15 test signs or scale, 11 test dates or periods and 34 match none of those patterns.

None of the 180 in-force documents mentions an invoice, a purchase order, a source document, a supporting document or an audit trail. The word stem approv appears in 13, every time about the committee's own approval, a rule's status, approved taxonomies or approved key names; authoriz appears in 2, both about shares authorized; receipt appears in 2, about cash collections and depository receipts. The rule for assets, DQC_0004, shows the style: it checks that assets equal liabilities plus equity within the filing and flags the gap.

That is the right job for a filing rulebook, and it marks the boundary this pilot tests. A machine check that sees only the record can catch a total that fails to foot or a value with the wrong sign. Whether a cost belonged to September, whether the person who changed a vendor's bank details had authority, or whether a second invoice was the same obligation needs evidence from outside the entry, which is exactly what a verified correction records. The rulebook also keeps growing, with 23 in-force rules dated 2026 and 24 in each of 2023, 2024 and 2025, so a pilot that uses rules as its baseline should name the rule set version.

Counting rules, exactly as the script applies them: each docs/DQC_US_* folder holds one rule document; status comes from its Status line, and a rule is in force when that line contains Final or Approved and neither Retired nor Removed. Each title takes the first class, in this order, with a keyword that appears in it as a whole word: sign or scale (negative, nonneg, sign, scale, scaling, percent), arithmetic (calculat, equal, less than, greater, exceed, subtotal, aggregat, roll forward, opening and closing, reconcil, cross calc, ratio, addition, breakdown, net of, gross), date or period (date, period, duration, year, time), tagging (axis, member, element, extension, dimension, concept, enumeration, line item, caption, deprecated, table, text block, abstract, symbol, placement), else other. A keyword counts only where it begins a word, at the start of the title, after any character that is not a letter, or at a capital letter inside a run-together name such as ScenarioAdjustmentMember. It may run on to the end of that word, so stems and plurals count (calculat matches Calculation, member matches Members), but a keyword inside another word does not (ratio in Operations, date in Consolidated, aggregat in Disaggregation), and sign must be the whole word. Shares are rounded half up once and total 100.0 percent:

What 180 public filing rules can check, and what they leave to evidence
Class of checkIn-force rulesShare of 180Example rule title
Tagging choice: element, axis or member7742.8%Axis with Inappropriate Members (DQC_0001)
Arithmetic or balance relationship4323.9%Element Values Are Equal (DQC_0004)
Sign or scale158.3%Negative Values (DQC_0015)
Date or period116.1%Context Dates After Period End Date (DQC_0005)
Other topic checks3418.9%Inappropriate Cash Flow Presentation (DQC_0145)

One difference between the arms: the matched design

Two arms that differ in one thing: whether the packets carry verified correction history. Everything else, from the model and its instructions to the number of packets, the companies, the periods and the compute budget, stays fixed, so a difference in detection can be traced to the history and not to volume or source.

Matching does the work a coin flip would do in a clinical trial. Elizabeth Stuart's 2010 review in Statistical Science describes matching as a way to approximate a randomized experiment with observational data by making the treated and control groups' covariate distributions similar. Here the covariates are the traits that make a packet easier or harder to judge, and balance is checked after matching rather than assumed. The protocol, fixed before any candidate data arrives:

One difference between the arms: the matched design
Design elementSpecificationWhat it rules out
UnitOne record packet: the entry or item, its source documents, postings and approvalsScoring loose lines with no evidence attached
Arm APackets with verified correction history: original, error found, correcting entry, finder, evidence, reviewer confirmationUnverified edits posing as corrections
Arm BMatched packets from the same companies and periods showing only the final stateComparing different companies or years
Matching keysCompany, workflow type, period, amount band, counterparty type and number of attached documents, with balance checked after matchingEasier packets landing in one arm
Frozen test setPackets from a later period that neither arm saw, each shown as it stood before any correctionTesting on what either arm learned from
LabelsSet by two qualified accountants working independently before any model runs; a third settles disagreementsLabels shaped by model outputs
RepeatsThree training or tuning runs per arm with fixed seeds published in advance; report the range, not the best runA lucky run read as an effect

Error classes the frozen test set must contain

The test set needs confirmed errors in every class the decision depends on, and clean look-alikes beside them, or the pilot cannot tell better detection from more flagging. Each row below names what a final-state record shows, what verified history adds and the evidence reviewers need before a label stands.

Accounting exception data for the test set should cover every row. The casebook is illustrative and should be adapted to the workflows a buyer's system will face. The look-alike row is the one most often skipped, and without it a false-alarm rate cannot be measured:

Error classes the frozen test set must contain
Error classWhat the final-state record showsWhat verified correction history addsEvidence reviewers need for the label
Duplicate paymentTwo paid invoices that look like separate billsThe later recovery or credit and the note tying both to one obligationBoth invoices, the order or contract, the bank record of the refund
Wrong periodAn accrual or expense booked in a plausible monthThe reclassifying entry and the service dates that drove itDelivery or service dates, the close calendar, the adjusting entry
Misclassified accountA balanced entry to a reasonable-looking accountThe reclass entry and the policy reference that decided itAccount policy, the item description, the reclass approval
Unauthorized changeA payment term or vendor bank detail that matches the master fileThe reversal and the finding that the approver lacked authorityDelegation of authority, the change log, the approver's role
Arithmetic or extension errorA line total that differs from quantity times priceThe corrected amount and who caught itSource invoice lines and the corrected posting
Missing supportA posting with no attached documentThe request for support and its outcome, supported or reversedThe document request log and the final evidence or reversal
Clean look-alikeA correct entry resembling one of the classes aboveNo correction, plus a reviewer's note confirming the entry was rightThe same evidence as the class it resembles, showing no error

Grading each flag: five verdicts and two independent reviewers

Score every output on every test packet with one of five verdicts, then count them separately by error class. Two qualified accountants assign the reference labels independently before any model output exists, a third settles disagreements, and agreement before that adjudication is reported as a result in its own right, since a test set its own reviewers dispute cannot grade a model.

At a glance

Grading each flag: five verdicts and two independent reviewers

The verdicts, from the strongest result to the weakest:

  • Hit with the right reason: the flag names the confirmed error class and points to the evidence that shows it.
  • Correct escalation: for unauthorized-change and missing-support packets, the output routes the packet to a person and names the missing authority or document; this counts as a hit, because the packet alone cannot prove the error.
  • Hit with the wrong reason: the packet is flagged, but the class or evidence named is one the reviewers reject; it counts toward recall and is also reported on its own line.
  • Miss: a confirmed error with no flag.
  • False alarm: a flag on a clean packet, reported as a share of all clean packets and broken out for clean look-alikes.

How many confirmed errors the test set needs

More than intuition suggests: detecting a 10-point recall gain with 80 percent power takes between 168 and 249 confirmed errors in the test set under this article's assumptions, and a 5-point gain takes 498 to 817. Because both arms score the same packets, the test compares the cases only one arm caught, which is a paired design.

The analysis uses McNemar's test on those discordant cases. Dietterich's 1998 comparison of statistical tests for classifiers, published in Neural Computation, found McNemar's test to have low type I error. The sizes below were computed exactly for this article: an exact two-sided binomial test on the discordant cases at an alpha of 0.05, with the decision computed in integers and power summed over every possible number of discordant cases. It uses no random numbers. The first size reaching 80 percent power also held at or above 80 percent for the next 25 sizes in every row.

Discordance is the share of error cases that exactly one arm catches; it is an assumption to replace with a dry run on reviewer-labeled packets before the test set is frozen. All inputs are Illustrative:

How many confirmed errors the test set needs
Recall gain for arm AConfirmed errors needed if 5% are caught only by arm BConfirmed errors needed if 10% are caught only by arm B
5 points498 (15% discordant)817 (25% discordant)
10 points168 (20% discordant)249 (30% discordant)
15 points92 (25% discordant)129 (35% discordant)
20 points61 (30% discordant)84 (40% discordant)
A two-by-two grid of paired test outcomes, both arms detect, only the correction-history arm detects, only the final-state arm detects, neither detects, with the two discordant cells highlighted as the cells the paired test compares, beside the error counts needed for 5, 10 and 15 point recall gains.
The paired comparison at the center of the pilot. Required error counts are Illustrative, computed exactly for 80 percent power at alpha 0.05 with 5 percent of errors caught only by the final-state arm.

Could arithmetic, leakage or matching explain a gain?

Each of the three can, and the design has to rule them out before a gain is credited to correction history.

Arithmetic first. The strongest objection to the whole idea is that correct arithmetic does not establish correct treatment or authorization, and its mirror image is that a gain concentrated in arithmetic and extension errors may be one any rule could deliver. Report recall by class, run a rule-based baseline that recomputes totals and signs, and credit correction history only with gains beyond that baseline, especially in the wrong-period, misclassification and unauthorized-change classes.

Leakage second. Kaufman, Rosset, Perlich and Stitelman define leakage in data mining as introducing information about the target that should not legitimately be available to mine from. In this pilot the obvious leak is a test packet that still contains its correcting entry or reviewer note; the subtler one is a near-duplicate of a training packet. Strip every field dated after the error from test packets, and remove near-duplicates across arms and test set before the freeze.

Matching third. If arm A's packets come from companies with stronger controls, any gain may reflect the companies rather than the history. Report standardized differences on every matching key after matching, and rerun the comparison within companies that contributed to both arms. Two smaller threats deserve a line in the report: arm A's packets are longer, so equalize the context or token budget, and errors are rare in real books, so report precision at the observed error rate as well as at the test set's enriched rate.

How results will be reported, and why none appear here

No results appear on this page, because the pilot needs licensed, de-identified records from companies that keep corrections, and no such run has been published. The report will carry the items below, in this order, whatever the result; the first is published before any candidate data arrives:

At a glance

How results will be reported, and why none appear here

  • The hypotheses, error classes, matching keys, test set size per class and the stopping rule, dated before candidate data arrives.
  • Reviewer agreement before adjudication, overall and by error class.
  • Recall by error class for each arm, the false-alarm rate on clean packets and on clean look-alikes, and precision at the observed error rate.
  • The two discordant counts, packets only arm A caught and packets only arm B caught, with the exact McNemar p-value.
  • The range across the three seeded runs per arm, and every excluded packet with its reason.
  • Null and negative results in the same detail as positive ones.

Requesting records that can run this test

A finance workflow dataset requested for this pilot should be specified by the test, not the volume. Name the workflows and error classes, the number of corrected packets and matched final-state packets sought, the later period to hold out for the test set, and the acceptance check a sample must pass: every corrected packet carries the original entry, the correcting entry, the evidence and a reviewer's confirmation.

SourceX sources datasets on request from businesses rather than holding them in stock, and the request form at sourcex.si/request-data takes scope, volume, format, timing and licensing needs. Before any record leaves a company, that company approves the scope and terms and signs; de-identification is done by SourceX directly or through its processing partner. A request starts a search, not a delivery: nothing here implies that suitable records exist today, that a sample will pass, or that correction history will improve any model.

The limits of this design are worth stating plainly. It tests detection, not correction; it says nothing about whether a model can draft the fix. Its rule analysis covers public filing rules, not the controls inside any company's ledger, and its sample sizes rest on assumed discordance. Accounting labels need qualified reviewers; this page is general information, not accounting advice.

Frequently asked questions

What is a finance dataset error detection pilot?

A controlled test of whether a particular kind of finance record helps an AI system find errors. This design compares matched packets with and without verified correction history and scores both on a frozen, reviewer-labeled test set, counting missed errors and false alarms by error class.

Why match records instead of comparing two datasets as delivered?

Because two datasets usually differ in company, period, workflow and volume as well as in correction history. Matching on those traits, and checking balance afterward, isolates the one difference the pilot is meant to measure.

How many errors does a finance detection test set need?

Usually more than a pilot budget assumes. Under this article's illustrative assumptions, detecting a 10-point recall gain with 80 percent power takes 168 to 249 confirmed errors, and a 5-point gain takes 498 to 817, all scored by both arms.

Can rule-based checks find the same errors?

Some of them. Of 180 in-force XBRL US Data Quality Committee rules, 43 test arithmetic or balance relationships and none refers to invoices, purchase orders or supporting documents. Rules catch errors visible in the record; wrong-period, misclassification and authorization errors need outside evidence.

Have results from this correction history test been published?

No. The page publishes the design, definitions, metrics and reporting rules only. Running it needs licensed, de-identified records from companies that keep their corrections, and every result will be reported in the order listed, including null results.

Sources

  • The XBRL US Data Quality Committee's rule repository at commit 5f8cd32 holds 185 U.S. rule documents with titles, statuses and release dates; 180 are in force under the stated rule. Source
  • Rule DQC_0004 tests that values in accounting-constant relationships, such as assets equal liabilities and equity, are consistent within the filing. Source
  • ISA 450 defines a misstatement as a difference in the amount, classification, presentation or disclosure of a reported item from what the framework requires, arising from error or fraud. Source
  • Stuart (Statistical Science, 2010) describes matching as approximating a randomized experiment by making treated and control covariate distributions similar. Source
  • Dietterich (Neural Computation, 1998) compared statistical tests for classifiers and found McNemar's test to have low type I error. Source
  • Kaufman, Rosset, Perlich and Stitelman (ACM TKDD, 2012) define leakage as introducing information about the target that should not be legitimately available to mine from. Source
  • SourceX's request page takes scope, volume, language, format, timing and licensing needs and says datasets are sourced on request, not held in stock. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies