Skip to content

AI uses for records

Spreadsheet audit benchmark for business: a protocol for silent errors

By SourceX Editorial · Updated

9 sections

Short answer

This is a test protocol, not a result. A spreadsheet audit benchmark for business workbooks should give a model real, rights-cleared workbooks that contain errors which leave totals plausible, such as omitted ranges, hard-coded overrides and numbers stored as text, plus clean workbooks as controls. The model must locate each error, explain the evidence and name the affected output. Grading recalculates the fix. No scores are reported.

Evaluation split

Spreadsheet audit benchmark

This page covers spreadsheet audit benchmark for business: a protocol for silent errors.Illustrative

Key takeaways

  • This page proposes a benchmark design and reports no model scores.
  • Test audit judgment: the model must find errors that leave a workbook looking correct and say which output they distort.
  • Run natural, seeded and clean tracks, and report them separately so seeding style and always-flag behavior cannot inflate results.
  • Split by company and hold out a whole task family, because templates repeat within a company.
  • Grade by applying the fix in a recalculating engine, never by reading cached values.

How could a benchmark test spreadsheet errors that do not affect appearance?

By grading whether a model finds the error, not whether it can edit the sheet. Each task hands the model a business workbook and one question: is anything wrong that changes a reported number, and if so, where? Some workbooks carry an error that was there naturally and later confirmed by the owner, some carry an error seeded by the benchmark team, and some are clean. A correct answer names the cells, the error class, the evidence and the output it distorts. A grader then applies the proposed fix in a calculating engine and checks that the key output moves to the verified value.

The hard part, and the point of the benchmark, is that every error in it leaves the workbook looking normal. There is no #REF!, no red cell and no obviously wrong total. This page sets out the proposed task, schema, grading and test protocol for review before any workbook is collected. It reports no model scores and no error rates, and any example below is illustrative.

Which errors leave a workbook looking fine?

Silent errors share one property: the formula still returns a number of the right size and sign. The classes below form the benchmark's taxonomy. Several have well-documented real cases. In the critique of Reinhart and Rogoff's public debt study, Herndon, Ash and Pollin reported a spreadsheet coding error that excluded five countries from an average, alongside selective data exclusion and unconventional weighting. A 2016 screen of supplementary files in genomics found gene-name errors, such as symbols converted to dates, in roughly one fifth of papers with Excel gene lists. Neither error announced itself on screen.

Which errors leave a workbook looking fine?
Error classHow it hidesGround truth neededFlagged by Excel's built-in rules?
Omitted rangeSUM stops a few rows short after rows were insertedCorrect range and corrected totalSometimes, as formulas that omit cells in a region
Hard-coded overrideA typed value replaces a formula in one cell of a columnOriginal formula and owner confirmationSometimes, as an inconsistent formula
Number stored as textA pasted figure is skipped by SUM but displays normallyCell list and corrected totalYes, as numbers formatted as text
Reference driftA copied formula points one row or column offIntended reference patternSometimes, as an inconsistent formula
Wrong lookup modeApproximate match returns a near neighbor on unsorted keysExpected matches for affected rowsNo
Double-counted subtotalA grand total sums detail rows and their subtotalsCorrect aggregation ruleNo
Unit or sign mixThousands mixed with units, or credits entered as positivesUnit and sign convention from the ownerNo
Silent type conversionCodes or IDs auto-converted to dates or numbers on importOriginal values from the source systemNo
Stale valueManual calculation or a broken external link leaves an old cached resultFreshly recalculated outputNo

What do existing spreadsheet benchmarks and checkers leave out?

Existing work tests either manipulation or pattern anomalies, not audit judgment on business workbooks. SpreadsheetBench, for example, built 912 instructions from online Excel forums with 2,729 test cases, and counts a solution as correct only if it passes several spreadsheet variants for the same instruction. That online-judge design is worth copying, but its tasks ask a model to do something to a sheet, not to decide whether a finished sheet is wrong.

Static checkers cover part of the gap. Microsoft documents background error-checking rules in Excel that flag formulas inconsistent with neighbors, formulas that omit cells in a region, numbers formatted as text and unlocked cells containing formulas, among others, while cautioning that passing them does not mean a sheet is free of errors. ExceLint, a research tool, finds formula errors by looking for breaks in a sheet's rectangular structure; its authors reported strong precision on a 70-workbook corpus and found the Reinhart and Rogoff range error, but they also stressed that recall is inherently hard to measure on real spreadsheets because nobody knows every error a real workbook contains.

That recall problem shapes this protocol. A benchmark built only from natural errors will undercount what a model misses, and a benchmark built only from seeded errors will reward spotting the seeding style. The design therefore uses both and reports them separately.

Which workbooks go in, and how are companies held out?

Workbooks come from operating businesses that have cleared the rights to use them for evaluation, with personal data removed or replaced before any annotator sees them. Each workbook belongs to a task family, such as budgets and forecasts, pricing and quoting sheets, inventory and reorder planners, project cost trackers and reconciliation schedules, and each family has a written description of its normal structure.

The split is by company, not by workbook. A company's workbooks share templates, naming habits and macros, so if one of its sheets is in the test set, none of its sheets may appear in development data. The test set is also split by task family, so at least one family is held out entirely to test transfer. Public corpora such as the Enron spreadsheet collection, described as over 15,000 business spreadsheets extracted from the company's email archive, are useful for piloting the taxonomy and seeding scripts but stay out of the test set because they are public and may already sit in model training data.

Three tracks run on the same workbook pool: natural errors confirmed by the owner, seeded errors inserted by a script and reviewed by a domain expert for realism, and clean workbooks that an expert audit found no material error in. Clean workbooks are essential; without them a model that always reports something would look perfect on recall.

Seeding follows written rules. Each seeded error must be one a domain expert has seen in practice, must change at least one key output beyond the tolerance, and must survive a blind check in which a second expert, shown the workbook without the seeding log, cannot tell it from the rest of the sheet by formatting alone. Errors that fail the blind check are removed rather than kept as easy items.

The proposed workbook correctness task schema

Every task is stored as one record with a frozen workbook file, instructions and a ground-truth block the model never sees. The fields below are the minimum needed to grade and to audit grading later.

  • Each finding the model returns must include: the cell or range, the error class from the published taxonomy, the evidence in one or two sentences, the key output it affects and a proposed fix.
  • A no-material-error answer is a valid response and is the correct one on clean workbooks.
The proposed workbook correctness task schema
FieldContentVisible to modelUsed for
task_id and versionStable ID, task family, track (natural, seeded, clean)ID onlyReporting by family and track
workbook_fileFrozen file with hash, calculation mode and engine notedYesReproducibility
instructionsAudit question, the outputs that matter, allowed toolsYesDefines what counts as material
key_outputsCells whose values decide the workbook's purposeYesImpact grading
ground_truth_errorsRange, error class, evidence, corrected formula or valueNoMatching findings
verified_outputsKey outputs after correction, from a recalculating engineNoExecution check
provenanceCompany code, rights basis, redaction log, seeding script versionNoHoldout and release control
expected_response_shapeFindings list or explicit no-material-error statementYesParsing and abstention

How does the grader decide a finding is right?

A finding counts when its range overlaps a ground-truth error, its class matches, and applying its fix moves every affected key output to the verified value within a stated tolerance. Location alone is not enough, because pointing at the right column with the wrong reason is a guess. Class alone is not enough either, because a model can name a likely error type without finding it.

Recalculation must use a real calculating engine. Reading cached values is not a check: openpyxl's documentation explains that its data_only option decides whether a formula cell returns the formula or the value stored the last time Excel read the sheet, so a grader that reads cached values sees whatever the last calculating application saved, not the effect of the fix. The grader opens the patched file in a fixed engine version, recalculates fully and compares outputs.

Findings that match no ground-truth error go to two expert reviewers before they are scored as false alarms. If both confirm a genuine error the team missed, it is added to ground truth in a new task version and the model gets credit; this keeps the benchmark honest about its own incomplete recall. Every such addition is logged with the date and reviewers.

Partial credit is limited and declared in advance. A finding with the right range and class whose fix leaves an output wrong scores as located but not fixed, and it is reported in its own column rather than blended into recall. A finding on a real but immaterial issue, such as an inconsistent number format that changes no key output, is neither rewarded nor penalized; the instructions define materiality as a change to any key output beyond the tolerance, so the model knows the rule it is graded by.

Which scores can be read as general audit ability?

Only results reported separately for held-out companies and the held-out task family should be read as general audit ability. A model that scores well on familiar templates and poorly on a new family has learned template habits. The report therefore shows every metric three ways: in-distribution companies, held-out companies and the held-out family, each with its own interval.

Results on the seeded track also need a check that the model is not detecting the seeding script. The protocol varies seeding locations and styles, includes seeded errors that experts judged indistinguishable from natural ones, and compares performance on seeded and natural errors of the same class. A large gap in favor of seeded errors is a warning that the seeded track flatters the model.

Materiality also has to transfer. A budget workbook and a reconciliation schedule have different key outputs, so the instructions for every task name the outputs that matter rather than leaving the model to guess. If a model performs well only when key outputs are named and poorly when it must infer them, that is a real limit worth reporting, and the protocol includes a small variant track where key outputs are withheld to measure it.

How many companies, which baselines and what kind of interval?

Defensible means enough held-out companies that a company-clustered interval is narrow enough to separate the baselines, plus baselines that a reader recognizes. The protocol fixes the minimum number of companies and workbooks per family before collection, from the precision needed on the smallest reported cell.

At a glance

How many companies, which baselines and what kind of interval

Three baselines run on every task: Excel's own error-checking rules with all rules enabled, a rule-based structural linter, and trained human reviewers given the same time budget. Models are reported with exact version, date, settings and tool access. Uncertainty comes from a bootstrap that resamples companies rather than workbooks, since errors in one company's sheets are correlated.

  • Primary metrics: recall of material errors on natural and seeded tracks, false-alarm rate on clean workbooks, and execution-verified fix rate.
  • Secondary metrics: class accuracy, impact-weighted recall using the size of each output distortion, and abstention quality on ambiguous sheets.
  • Pre-registration items: taxonomy version, seeding script version, matching tolerance, tie-breaking rules, holdout list, baseline settings, minimum sample per cell and the exact reporting tables.
  • Release conditions: rights confirmed for each workbook, redaction log reviewed, test files kept private with access logging, and a refresh plan for when a test set is exposed.

Limits of the design, and where real workbooks come from

The benchmark measures error finding in the families it samples, under its taxonomy, with its tolerance. It does not measure whether a model can build a correct workbook, explain business logic it was not given, or judge errors that depend on facts outside the file. Natural-error ground truth depends on owners confirming errors, which means some real errors will stay unlabeled, and the expert review of unmatched findings narrows but does not close that gap.

The scarce input is rights-cleared business workbooks with owner-confirmed errors and corrections. Buyers who need such workbooks can request them through SourceX, which sources each dataset from U.S. businesses on request instead of holding stock; the request-data form takes the scope, volume, format, timing and licensing needs, such as evaluation-only use.

Frequently asked questions

What does the workbook audit protocol specify?

A published error taxonomy, rights-cleared workbooks split by company and task family, natural, seeded and clean tracks, a task schema with hidden ground truth, an execution-based grader, recognized baselines, company-clustered uncertainty and a pre-registration of all of it before scoring.

Why include clean workbooks with no errors?

Without them, a model that always reports a problem scores perfectly on recall. Clean workbooks measure false alarms, which matter in practice because an auditor that flags everything wastes the reviewer's time and gets ignored.

Can Excel's built-in error checking serve as the benchmark?

It is a useful baseline, not a benchmark. Microsoft's rules catch some silent errors, such as numbers stored as text and formulas that omit cells, but they do not judge lookup modes, double counting, unit mixes or stale values, and Microsoft cautions that a sheet passing them can still contain errors.

Why not use public spreadsheet corpora for the test set?

Public corpora such as the Enron spreadsheets are valuable for piloting, but they may already be in model training data, which would inflate scores. The test set uses private, rights-cleared workbooks that stay unpublished.

Have any models been scored on this benchmark?

No. This page describes a proposed task and test protocol only. Scores will mean something only after workbooks are collected under the stated rights conditions, ground truth is verified and the analysis plan is preregistered.

Sources

  • SpreadsheetBench contains 912 instructions from online Excel forums and 2,729 test cases, and counts a solution correct only if it passes multiple spreadsheet variants per instruction. Source
  • Microsoft documents Excel's error-checking rules, including formulas inconsistent with others in the region, formulas that omit cells in a region, numbers formatted as text and unlocked cells containing formulas, and cautions that the rules cannot make a worksheet error-free. Source
  • Herndon, Ash and Pollin's critique of Reinhart and Rogoff reported coding errors, selective exclusion of available data and unconventional weighting; the spreadsheet error excluded five countries from the computation. Source
  • A 2016 screen of supplementary files found gene-name errors, such as symbols converted to dates, in roughly one fifth of papers with Excel gene lists. Source
  • ExceLint's authors evaluated it on a 70-workbook benchmark corpus and a professionally audited spreadsheet, reported high median precision and recall, and noted that recall is inherently difficult to measure on real-world spreadsheets. Source
  • openpyxl's data_only option controls whether formula cells return the formula or the value stored the last time Excel read the sheet. Source
  • The Enron spreadsheet corpus contains over 15,000 business spreadsheets extracted from the Enron email archive. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies