AI uses for records
Document OCR error evaluation: is it the OCR or the reasoning?
By SourceX Editorial · Updated
10 sections
Short answer
To separate OCR errors from later reasoning errors, answer every question twice with the model, prompt and retrieval held fixed: once on OCR output and once on a verified transcription of the same regions. Failures that disappear with verified text are OCR-caused; failures that persist belong to reasoning, the question or the expected answer. Tie each case to its file, page, region and OCR confidence.
Document stack
Document OCR error
Key takeaways
- Attribute failures with paired runs that change only the text the reasoning step sees: OCR output in one arm, verified transcription in the other.
- Compute error rates on the evidence regions an answer depends on, not only page averages, and use a structural metric such as TEDS for tables.
- Arms that inject OCR errors only inside or only outside the evidence regions show whether the errors that matter are where the answer lives.
- Record every evidence region, including footnotes, stamps and linked files; a missing region is its own verdict, not a reasoning error.
- The verified transcription is an oracle with its own error rate; double-key a sample and exclude evidence verifiers cannot agree on.
How do you separate OCR errors from later document reasoning errors?
Run every question twice with everything held fixed except the text the reasoning step receives: once on the OCR output, once on a verified transcription of the same page regions. If the answer is wrong on OCR text and right on verified text, the OCR stage caused the failure. If it is wrong on both, the reasoning step, the question or the expected answer is at fault.
This is a counterfactual substitution test, and it is the cleanest way to stop arguing about blame in a document pipeline. The model, prompt, retrieval settings, tool definitions and decoding parameters stay identical between the two runs. Only the upstream text changes, so any change in the downstream answer is attributable to that text.
The research community uses the same logic at benchmark scale. OHR-Bench, from the OpenDataLab team, evaluates retrieval-augmented generation on human-verified ground-truth structured data and on the outputs of real OCR systems, then adds perturbed versions with what it calls semantic noise and formatting noise at mild, moderate and severe levels. The point for your own evaluation is the design, not their leaderboard: ground truth in one arm, OCR in the other, and the same downstream system scoring both.
What a benchmark cannot do is tell you which of your own documents fail and why. That needs a per-question record that ties each answer back to the exact file, page and region it depends on, which is the asset this page lays out.
Why won't a page-level character error rate tell you which answers broke?
Character and word error rates measure how far OCR text is from a reference, not whether the answer that depends on it survived. A page can score well overall while the one digit that decides the answer is wrong, and a page full of mangled boilerplate can still produce the correct answer.
Use the standard metrics, but compute them on the evidence regions as well as on the whole page. JiWER, a widely used Python package, computes word error rate and character error rate from the minimum edit distance between reference and hypothesis strings. The ocreval toolkit, a modern port of the ISRI Analytic Tools for OCR Evaluation with UTF-8 support, offers 17 tools for character and word accuracy and related reports. Either works for text regions.
Tables need a structural measure because a perfectly read cell in the wrong column is still a wrong value. The PubTabNet project released a tree-edit-distance-based similarity metric, TEDS, that compares a predicted table's HTML structure and content against ground truth. OmniDocBench evaluates document parsing across text, tables, formulas and reading order, using normalized edit distance for text and TEDS for tables, and its annotations include the reading order of page components.
The decision here is which upstream numbers to store per question: region-level character error rate for text evidence, TEDS or cell-level exact match for table evidence, and a reading-order check when the answer depends on which line follows which. Page averages go in a separate report; they explain trends, not individual failures.
What should the per-question record in an OCR error propagation evaluation hold?
Each question in the evaluation gets one record that holds its evidence, both input versions and both outcomes. The table below lists the fields. Values in the example column are illustrative.
| Field | What it records | Illustrative value | Why the attribution needs it |
|---|---|---|---|
| Case and question | Stable case ID, the question as asked, the expected answer and who confirmed it | Q-0412; quantity received for line 3; 16 units; confirmed by AP clerk | Without a confirmed answer, a disagreement cannot be scored at all |
| Source file and version | File name, content hash, page count and the date the version was captured | inv-7731.pdf; hash recorded; 2 pages | A re-scanned or amended file silently changes both arms |
| Evidence regions | Every page and bounding box the answer depends on, with region type | p1 table row 3 col Qty; p2 footnote | Attribution is only meaningful region by region |
| Layout and table context | Column headers, merged cells, footnote markers and reading order around each region | Header Qty Shipped above cell; footnote marker on row 3 | A correctly read value under the wrong header is a structural error |
| OCR output and engine | Text returned for each region, engine name and version, settings | Region text 18; engine and version logged | Results from different engines or versions are not comparable |
| OCR confidence | Word or block confidence for each evidence region, on the engine's own scale | Word confidence 61 on a 0 to 100 scale | Low-confidence evidence is where OCR-caused failures cluster |
| Verified transcription | Human-checked text for the same regions, verifier, and any disagreement | Region text 16; two verifiers agreed | This is the oracle arm; its quality caps the whole test |
| Cross-file links | Identifiers that connect this document to others the answer needs | PO number linking invoice to receiving record | Many business answers need two or more files |
| Run outcomes | Answer, score and trace for each condition, with model and prompt versions | OCR arm wrong; verified arm right | The verdict is computed from these two cells |
Which paired runs isolate the OCR stage from the reasoning step?
Two arms are the minimum; four make the verdict much easier to trust. The extra arms test whether errors inside the evidence region matter more than errors elsewhere, which is the core claim you are trying to prove.
Checklist · 3 items
- Keep the serialization identical across arms. If arm A emits Markdown-like tables and arm B emits plain text, you are testing formatting as much as recognition, which OHR-Bench treats as a separate kind of noise.
- Fix the random seed and temperature, and repeat each arm at least a few times if the model is not deterministic; record disagreement between repeats as its own number.
- Re-run retrieval in each arm rather than reusing arm A's retrieved chunks, because OCR errors can change what gets retrieved before reasoning starts.
0 of 3 ticked. Tick each item as you go.
| Condition | Text given to the reasoning step | What it isolates | How to read a change |
|---|---|---|---|
| A. Production | OCR output for the whole document, as the live pipeline produces it | The system as users experience it | The baseline every other arm is compared with |
| B. Verified | Verified transcription for every page, same layout serialization as A | Reasoning with OCR removed | A fails and B passes means OCR caused the failure |
| C. Evidence-only errors | Verified text, with the OCR output pasted back into the evidence regions only | The effect of errors where the answer lives | C fails like A: the evidence-region errors are sufficient to explain it |
| D. Off-evidence errors | OCR text everywhere except the evidence regions, which stay verified | Distraction from errors outside the answer | D fails: noise elsewhere misled retrieval or reasoning |
| E. Image input (optional) | The page image sent directly to a multimodal model, no OCR step | Whether bypassing OCR changes the result | Only comparable when the same model family handles both inputs |
How do you read the verdict: OCR-caused, reasoning-caused or something else?
The verdict comes from a small decision table over arms A and B, refined by C and D. Assign it automatically, then have a reviewer confirm every case that lands outside the two clean rows.
- OCR error types worth separating: character substitution (6 read as 8), dropped or merged tokens, table structure errors (value shifted to the next column or row), reading-order errors (two columns interleaved), missing regions (small print, stamps, handwriting) and version mismatch (the OCR ran on an older scan).
- Report the share of failures in each verdict row, with the denominator, and a confidence interval when the case count is small. A finding such as most failures being OCR-caused is only useful if it says out of how many cases.
| Arm A (OCR) | Arm B (verified) | Verdict | Next check |
|---|---|---|---|
| Correct | Correct | No failure | Sample a few for lucky guesses where the evidence region was misread |
| Wrong | Correct | OCR-caused | Classify the OCR error type and confirm with arm C |
| Wrong | Wrong, same answer | Reasoning-caused | Inspect the trace; then re-check the question and expected answer |
| Wrong | Wrong, different answer | Both stages contribute | Fix the reasoning issue first, then re-score the OCR effect |
| Correct | Wrong | Oracle or question problem | Re-verify the transcription and the expected answer before anything else |
| Unanswerable in A | Correct | Missing region | OCR dropped a footnote, stamp or attachment the answer needed |
Which image, table or attachment holds the evidence the answer depends on?
Before scoring, list every region the expected answer depends on, including regions in other files. If an evidence region is missing from the case record, a wrong answer cannot be attributed and should be excluded rather than guessed.
Business answers rarely live in one text block. A received quantity can depend on a table cell, a footnote that says the line was partially shipped, a handwritten correction on a delivery note and a separate receiving record linked by PO number. Each of those is a distinct evidence region with its own OCR risk, and each can fail independently.
Mark the region type, because the error profile differs. Machine-printed body text tends to fail on characters; tables fail on structure; footnotes and stamps fail by disappearing; handwriting fails on ambiguity that even human verifiers dispute. Engines expose enough to do this: Tesseract's hOCR and TSV outputs give a bounding box and a confidence value for each word, and Amazon Textract returns blocks with a Geometry object holding a bounding box and polygon, plus a Confidence score from 0 to 100.
Do not compare confidence numbers across engines as if they were the same scale. Use them within one engine and version to rank regions by risk, and to check whether low-confidence evidence regions really do account for more OCR-caused failures in your data.
How do the arms attribute failures in an illustrative billed-versus-received check?
The following walk-through is illustrative; the case counts are invented to show the method and are not results. Suppose an evaluation asks whether the quantity billed on each invoice line matches the quantity received, using scanned vendor invoices and receiving records linked by purchase order number.
Out of an illustrative 120 questions, arm A gets 21 wrong and arm B gets 10 wrong. Of the 21 arm A failures, 13 pass in arm B (OCR-caused), 7 fail with the same answer in arm B (reasoning-caused) and 1 fails with a different answer (both). Two questions pass in A and fail in B; both trace to a verified transcription that mis-keyed a line number, so they go back for re-verification rather than into the totals.
- Exception to note in the write-up: Q-0468 is not a model failure of either kind. Ambiguous source evidence should be reported as its own count so it is not silently absorbed into an error rate.
- Next step in this illustrative case: the table-structure errors suggest testing a layout-aware parser in arm A, while the credit-memo misses point to retrieval across linked files, a reasoning-side fix that better OCR would not touch.
| Case | Evidence regions | What happened | Verdict |
|---|---|---|---|
| Q-0412 | Invoice p1 table cell; receiving record line 3 | OCR read 16 as 18 at low word confidence; reasoning flagged a mismatch that does not exist | OCR-caused, character substitution |
| Q-0417 | Invoice p2 footnote; receiving record | Footnote saying the line shipped in two parts was absent from OCR output | Missing region |
| Q-0433 | Invoice p1 table, two-line description | Quantity shifted into the unit column after a wrapped description | OCR-caused, table structure |
| Q-0450 | Invoice p1; credit memo in a second file | Both arms ignored the linked credit memo although its text was correct | Reasoning-caused |
| Q-0468 | Delivery note with handwritten correction | Two verifiers disagreed on the corrected digit | Excluded as ambiguous |
When is a verified transcription not a fair stand-in for the page?
The verified arm assumes that a clean transcription carries everything the page does. When meaning lives in layout or marks rather than characters, that assumption breaks and the oracle itself becomes a source of error.
Check boxes, strike-throughs, highlights, stamps, signatures and initials often decide a business answer, and a plain transcription either drops them or encodes them in a way the production pipeline never would. Decide how each mark is serialized, apply the same rule in every arm, and record the rule with the dataset.
Image-native models complicate the comparison further. If the production system sends page images straight to a multimodal model, there is no separate OCR stage to blame, and arm B tests a different system. Use arm E in that case and frame the question as image input versus verified text input, not OCR versus reasoning.
Finally, the oracle needs its own quality bar. Double-key a sample of evidence regions, report agreement, and treat any region two careful verifiers cannot agree on as ambiguous evidence rather than ground truth.
Which signals mean the test should shrink or go to a records owner?
Narrow the evaluation when attribution stops being clean, and bring in the people who own the records when the expected answers themselves are in doubt.
- More than a handful of cases land in the oracle-or-question row: the expected answers need review by someone who works with these documents, not more model runs.
- Verdicts change between repeated runs of the same arm: the reasoning step is too unstable for single-run attribution; repeat and report agreement first.
- Most failures depend on cross-file links: the test is now about record linkage, and the case design should be rebuilt around linked document sets.
- Evidence regions are mostly handwriting or degraded faxes: verifier agreement will be low, so split these into a separate slice with its own reporting.
- Documents contain personal or regulated fields: route the dataset through a privacy review before any verifier sees the pages.
- The production pipeline changes OCR engine or version mid-study: stop, re-run arm A on the new version and keep the two result sets apart.
Where do documents with verifiable evidence regions come from?
The hard part of this evaluation is not the model runs; it is a set of real business documents with linked records and evidence regions someone can verify. Public document benchmarks rarely carry the cross-file links that business answers depend on.
For this kind of evaluation set, name the document types, the linked systems of record and the evidence-region granularity in SourceX's request-data form, together with the scope, volume, language, format, timing and licensing needs it asks for. SourceX holds no document catalog and looks for matching records at U.S. businesses only after such a request. The supplying company approves the scope and terms and signs first, and the preparation that follows, de-identification included, is done by SourceX through approved processing routes under that agreement.
Frequently asked questions
What is OCR error propagation in a document AI pipeline?
It is the way a recognition mistake, such as a misread digit or a dropped footnote, carries into retrieval and reasoning and produces a wrong final answer. Measuring it means comparing answers on OCR text with answers on verified text for the same regions, with everything else held fixed.
Is character error rate enough to evaluate OCR for an agent or RAG system?
No. Character error rate summarizes how far the text is from a reference, but one wrong character in an evidence region can flip an answer while many errors elsewhere change nothing. Compute it per evidence region and pair it with answer-level outcomes.
How many questions does an OCR error propagation evaluation need?
Enough that each verdict row you care about has a stable count. Start with a few hundred questions spread across document types and region types, report every share with its denominator, and add cases where the confidence intervals are too wide to act on.
Can a multimodal model that reads images directly skip this test?
It changes the test rather than removing it. Compare image input against verified text input for the same questions; if image input fails where verified text passes, the recognition happening inside the model is still the cause.
Who should verify the transcriptions used as the oracle?
People who know the document type, such as the accounts payable or records staff who process those forms, checked by a second verifier on a sample. Their agreement rate belongs in the report, because it limits how confident any attribution can be.
Sources
- OHR-Bench evaluates the cascading impact of OCR on retrieval-augmented generation using human-verified ground truth, real OCR outputs and perturbed data with semantic and formatting noise at mild, moderate and severe levels. Source
- JiWER computes word error rate and character error rate from the minimum edit distance between reference and hypothesis strings. Source
- ocreval is a set of 17 tools for measuring OCR output, a modern port of the ISRI Analytic Tools for OCR Evaluation with UTF-8 support. Source
- PubTabNet annotates table images with HTML and released the tree-edit-distance-based similarity metric TEDS for table recognition. Source
- OmniDocBench evaluates document parsing across text, tables, formulas and reading order, with normalized edit distance and TEDS among its metrics and reading-order annotations for page components. Source
- Tesseract's hOCR and TSV outputs report a bounding box and a confidence value for each recognized word. Source
- Amazon Textract returns Block objects with a Geometry location (bounding box and polygon) and a Confidence score between 0 and 100. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.