Systems and records
Scanned document revision benchmark: can AI pick the final approved scan?
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
To test whether an AI recognizes the final approved scan, give it a set of candidate scans of the same document, only one of which is approved and in force, and score three things: did it pick the right file, did it point to the page region that proves approval, and did it abstain when no candidate qualifies. Build each item from real revision histories with hard distractors such as unsigned copies and superseded revisions.
Document stack
Scanned document revision
Key takeaways
- Treat version recognition as a selection task across candidate scans, not as reading one page.
- Define draft, approved, effective and superseded before labeling, and put a question date in every item.
- Record the page region that proves approval so grounding can be scored, along with OCR confidence for that region.
- Hard distractors such as unsigned finals, rescans of old revisions and amended copies decide whether the benchmark means anything.
- Score selection, grounding and abstention separately, and report superseded picks as their own error.
How do you test whether an AI recognizes the final approved scan?
You test it as a selection task, not a reading task. Each test item holds two or more scans that claim to be the same document, a question such as which of these is the approved version in effect on a stated date, and a gold answer that names one file, the page region that proves it, and the other records a reviewer used to decide. A model passes an item only when it selects the right file and grounds the choice in the right evidence.
Most document AI benchmarks ask a model to read one page and extract a value. DocVQA, a widely used benchmark, defines its questions on single document images and asks for answers that appear in the page text. That tests reading. A scanned document revision benchmark tests judgment across files: the model has to compare revision letters, signature blocks, stamps and dates, notice that one copy is a draft and another is superseded, and sometimes admit that none of the candidates is approved.
This guide gives the definitions a labeler needs, the regions of a scan that carry version evidence, a field-level specification for the test set, a distractor catalog, a scoring rule, an illustrative worked item and the cases where a single right answer does not exist.
Final, latest and approved are three different labels
Decide what the gold label means before anyone labels a file, because the newest scan is often not the approved one. A later scan can be a marked-up working copy, a rescan of an older revision pulled from a binder, or a draft circulated for comment after the approved version was issued. Define four states and apply them per file: draft (not approved), approved (an authorized person signed or stamped it), effective (approved and in force on the question date), and superseded (approved once, replaced by a later approved revision).
Approval needs evidence of who approved, when, and in what capacity. In FDA-regulated electronic records, 21 CFR 11.50 requires signed records to show the signer's printed name, the date and time of signing, and the meaning of the signature, such as review, approval, responsibility or authorship. Scanned paper rarely meets that standard, but the same three elements make a good labeling rule for any industry: a signature without a stated meaning or date is weaker evidence than one with both.
Write the question date into every item. The same set of scans has different correct answers on different dates, and a test that omits the date rewards models that simply pick the highest revision letter.
| Label | Definition used in the test set | Typical evidence on the scan | Common false signal |
|---|---|---|---|
| Draft | Not approved by an authorized person | Draft watermark, blank signature block, for comment stamp | A clean copy with no watermark that was never signed |
| Approved | Signed or stamped by an authorized approver with a date | Completed signature block, approval stamp, approval date in a revision table | A reviewer initial that records review, not approval |
| Effective | Approved and in force on the question date | Effective date field, release stamp, distribution record in another file | An approval date that precedes a later approved revision |
| Superseded | Approved once, replaced by a later approved revision | Superseded or obsolete stamp, later revision in the set, change notice | An old copy with no stamp because nobody marked it |
Which regions of a scan carry the version evidence?
Version evidence clusters in a few places, and the test set should record each one as a page number plus a bounding box so that grounding can be scored. On engineering drawings it sits in the title block and revision table; on procedures and specifications it sits in the header, footer and approval page; on contracts and forms it sits in the signature block and any handwritten amendment in the margin.
At a glance
Which regions of a scan carry the version evidence
Record OCR confidence for the words inside each evidence region, because a model that misreads a revision letter fails for a reading reason, not a judgment reason, and the error analysis should be able to tell them apart. The ALTO XML schema version 4.4 defines a word confidence attribute between 0 (unsure) and 1 (sure), and the hOCR specification defines an engine-specific x_wconf property where higher values express higher confidence. Store whichever your OCR engine emits, with the engine name, since the scales are not comparable across engines.
Capture scan resolution with each file. Tesseract's documentation says the engine works best on images of at least 300 dpi, so a low-resolution fax copy is a fair test of robustness but should be tagged so results can be split by image quality.
- Title block or document header: document number, title, revision letter or number, date.
- Revision table or change history: each revision, its description, date and approver.
- Signature or approval block: signer name, role, date, and the stated meaning of the signature.
- Stamps and watermarks: draft, approved, released, superseded, controlled copy, uncontrolled copy.
- Handwritten marks: redlines, margin amendments, initials, struck-through values.
- Footer and page count: page x of y, which exposes missing approval pages.
The document version selection test set, field by field
Each test item is one record with the fields below. The specification is deliberately strict about provenance and cross-file links, because the hardest items are the ones where the scan alone cannot settle the answer and the labeler relied on a change order, a document control log or an email approval. Federal digitization rules point the same way: 36 CFR 1236.54 requires unique file names and record IDs for each image file and relation metadata for records made of multiple parts, which is the minimum a test set needs to keep candidates and their evidence linked.
Assemble candidate sets from whole document histories rather than from random pairs. Pull every scan that shares a document number, add copies found under a different file name by matching titles and content, and only then decide which belong in the item. Split train and test by document family, not by item, so that no revision history appears on both sides; otherwise a model can memorize which copy of a given procedure was approved instead of learning to read approval evidence.
Freeze each item once adjudicated and give the set a version number and date. If a later control log changes a gold answer, issue a new set version and keep the old one, so results stay comparable over time.
| Field | What it holds | Validation check | Why it matters |
|---|---|---|---|
| item_id and question_date | Stable item ID and the date the question asks about | Unique ID; date present and in ISO format | The same candidates have different answers on different dates |
| candidate_files | Two or more scan IDs, each with page count, dpi and source system | Every ID resolves to a file; dpi recorded | Defines the choice set and supports quality splits |
| gold_selection | One candidate ID, or none_qualifies | Exactly one value; none_qualifies allowed | Tests abstention as well as selection |
| per_file_state | Draft, approved, effective or superseded for every candidate | One state per candidate; one effective file at most unless the item is a declared exception | Lets you score near misses and diagnose errors |
| evidence_regions | Page and bounding box for each region the labeler used | Box lies within page bounds; at least one region for the gold file | Scores grounding, not just the final pick |
| evidence_type | Revision table, signature block, stamp, watermark, handwritten mark or external record | Value from a closed list | Shows which evidence types models miss |
| ocr_layer | Engine name, version and word confidence for each evidence region | Engine and version present; confidence on the engine's own scale | Separates misreading from misjudging |
| cross_file_links | IDs of change orders, control logs or approval emails the labeler consulted | Each ID resolves; empty only when the scan alone decides | Marks items that need context beyond the scan |
| distractor_tags | Which distractor families each wrong candidate represents | Values from the distractor catalog | Lets you report accuracy by difficulty |
| label_provenance | Labeler ID, adjudicator ID, date and agreement status | Two independent labels; disagreements adjudicated | Makes the gold answer auditable |
Which distractors make the benchmark hard enough to matter?
A test set built only from clean pairs of an old and a new revision is too easy, because a model can pick the higher revision letter and score well. Real document stores contain copies that look final and are not, and the benchmark should sample each family below in proportions the team declares in advance. Tag every wrong candidate with its family so results can be broken down.
| Distractor family | What the wrong candidate looks like | What the model must notice | Where it comes from |
|---|---|---|---|
| Unsigned final | Identical content to the approved file but the signature block is blank | The missing approval, not the content | Copies printed before signing |
| Superseded approved | Fully signed and stamped, but a later approved revision exists | The later revision in the set or a superseded stamp | Old binders and shared drives |
| Marked-up working copy | Newest scan, covered in redlines | Handwritten changes mean it is a proposal | Review cycles scanned for the record |
| Rescan of an older revision | Recent scan date, older revision letter | Scan date is not revision date | Bulk back-file scanning projects |
| Wrong approver | Signed, but by a role that cannot approve this document type | Signer role against the approval rule | Delegation errors and stand-in signatures |
| Amended final | Approved file with a later handwritten margin change and initials | Whether the amendment was itself approved | Field changes recorded on paper |
| Missing approval page | Content pages present, approval page absent | Page x of y and the gap | Partial scans and split PDFs |
| No qualifying candidate | Every candidate is a draft or superseded | That the right answer is to abstain | Requests where the approved copy was never scanned |
How should a revision benchmark score answers?
Score selection, grounding and abstention separately and report each, because a single accuracy number hides the failure that matters most in production: confidently choosing a superseded copy. Selection is correct when the predicted file equals the gold selection. Grounding is correct when the model's cited region overlaps a gold evidence region on the gold file by at least the threshold your team declares before running the test. Abstention is scored on none_qualifies items and on items where the model declines.
Report selection accuracy split by distractor family, by image quality and by whether the item needed cross-file links. Weight errors by consequence if the use case warrants it: picking a superseded approved revision is usually worse than abstaining, and the scoring rule should say so in writing before any model is run.
Do not reuse reading metrics as the headline. DocVQA scores extracted answer strings against page text with a normalized string similarity, which suits extraction. A version selection item has a categorical answer, so exact match on the file ID plus a grounding check is the honest measure.
- Selection: predicted file ID equals gold_selection.
- Grounding: cited region overlaps a gold evidence region on the gold file above the declared threshold.
- Abstention: correct when the model declines on a none_qualifies item; counted as an error when it declines on an item with a valid answer.
- Severity: superseded or draft picks reported apart from other errors.
- Splits: distractor family, dpi band, OCR engine, cross-file link required or not.
Worked example (illustrative): revision C, revision D and an unsigned rescan
This item is invented for illustration and does not come from any company's records. A manufacturer's inspection procedure exists as three scans. File 1 is revision C, signed by the quality manager with the meaning approved and a date. File 2 is revision D, identical in layout, with revision D in the title block and a completed revision table entry, but the approval signature block is blank. File 3 was scanned most recently; it is revision C again, pulled from a work-cell binder, with a handwritten margin change to one inspection interval and initials but no date.
The question asks which file is the approved version in effect on a date after all three scans exist. The gold answer is File 1. File 2 is newer but unapproved; the labeler confirmed through the document control log, recorded as a cross-file link, that revision D was still in review on the question date. File 3 is an amended copy of revision C whose margin change was never approved. Evidence regions are the signature block on page 1 of File 1, the blank signature block on File 2, and the margin note on page 3 of File 3.
A model that answers File 2 has fallen for the newest revision letter. A model that answers File 3 has trusted the most recent scan date. A model that answers File 1 but cites only the title block gets selection right and grounding wrong, which is still useful to know.
The exception: had the control log shown that revision D was approved electronically and the paper copy simply never signed, the gold answer would change to File 2 and the item would carry evidence_type external record. That is why the cross_file_links field exists.
When a single right answer does not exist
Some document families have more than one valid approved version at once, and forcing a single gold answer into those items teaches the model something false. Site-specific variants of the same procedure, documents approved per customer or per jurisdiction, and controlled copies issued to different departments can all be simultaneously effective. Either narrow the question so one answer is correct, for example by naming the site, or exclude the family from the test set.
Exclude items where the approval lived only in a system you cannot show the labeler, such as an e-signature platform that was not exported, unless the item is explicitly built to test abstention. Send to a records management specialist any item where the labelers disagree after adjudication, where the approval authority depends on an internal delegation policy, or where the document is regulated and the approval rule comes from a regulation rather than company practice.
The limits of the method are worth stating on the results page: a benchmark built from one industry's documents measures that industry's conventions, revision schemes vary between letters, numbers and dates, and a model that does well on title blocks may still fail on forms where approval is a stamp.
Sourcing scanned revision histories with their approval trail
The most useful raw material is a document control archive where every revision was scanned and the control log, change orders and approval records survive alongside the images. Back-file scanning projects, quality system archives and engineering document stores often hold exactly this, including the messy copies that make good distractors. Having access to those scans is not the same as having the right to license them: customer drawings, supplier documents and contracts with confidentiality clauses each need their own review, and removing names does not create a licensing right.
AI teams can commission such archives through SourceX, which seeks them out at U.S. businesses only after a request specifies scope, volume, format, timing and licensing terms. The supplying company approves scope and terms and signs before any records are delivered, and SourceX handles rights review, preparation and delivery between the two.
Frequently asked questions
What should a document version selection test set include?
Each item needs a question date, two or more candidate scans, one gold selection or an explicit none_qualifies answer, a state label for every candidate, the page regions that prove approval, and links to any outside records the labeler used. OCR confidence and scan resolution per evidence region let you separate reading errors from judgment errors.
Which page region holds the evidence that a scan is the approved version?
Usually the signature or approval block, the revision table and the title block, with stamps and watermarks as supporting evidence. Handwritten margin changes and missing approval pages are the regions models most often overlook, so record them as evidence regions when they decide an item.
Why not just pick the scan with the highest revision letter?
Because the highest revision may still be in review, and the most recent scan may be a rescan of an older revision. A benchmark that rewards the highest letter measures sorting, not approval recognition, which is why distractors such as unsigned finals and rescans belong in the set.
How many labelers does each item need?
Two independent labelers with adjudication of disagreements is a sound default, and the adjudicator's decision should be recorded in the item's provenance. Items that still divide reviewers after adjudication are better excluded or sent to a records specialist than kept with a contested gold answer.
Should the model be allowed to say none of the files is approved?
Yes, and it should be scored on that ability. Real requests often arrive with only drafts and superseded copies, and a model that always picks something will confidently return the wrong document in exactly those cases.
Sources
- Signed electronic records must show the signer's printed name, the date and time of signing, and the meaning of the signature, such as review, approval, responsibility or authorship. Source
- Federal digitization metadata rules require a unique file name and record ID for each image file and relation elements for multi-part records. Source
- ALTO 4.4 word confidence (WC) is a value between 0 (unsure) and 1 (sure) for the OCR of a string. Source
- The hOCR x_wconf property is an OCR-engine-specific confidence for the contained substring, where higher values express higher confidence. Source
- Tesseract works best on images with a resolution of at least 300 dpi. Source
- DocVQA is a visual question answering dataset defined on single document images, with answers taken from the document text and scored by normalized string similarity. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.