Skip to content

Systems and records

Scanned document revision benchmark: can AI pick the final approved scan?

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

To test whether an AI recognizes the final approved scan, give it a set of candidate scans of the same document, only one of which is approved and in force, and score three things: did it pick the right file, did it point to the page region that proves approval, and did it abstain when no candidate qualifies. Build each item from real revision histories with hard distractors such as unsigned copies and superseded revisions.

Document stack

Scanned document revision

This page covers scanned document revision benchmark: can AI pick the final approved scan.Illustrative

Key takeaways

  • Treat version recognition as a selection task across candidate scans, not as reading one page.
  • Define draft, approved, effective and superseded before labeling, and put a question date in every item.
  • Record the page region that proves approval so grounding can be scored, along with OCR confidence for that region.
  • Hard distractors such as unsigned finals, rescans of old revisions and amended copies decide whether the benchmark means anything.
  • Score selection, grounding and abstention separately, and report superseded picks as their own error.

How do you test whether an AI recognizes the final approved scan?

You test it as a selection task, not a reading task. Each test item holds two or more scans that claim to be the same document, a question such as which of these is the approved version in effect on a stated date, and a gold answer that names one file, the page region that proves it, and the other records a reviewer used to decide. A model passes an item only when it selects the right file and grounds the choice in the right evidence.

Most document AI benchmarks ask a model to read one page and extract a value. DocVQA, a widely used benchmark, defines its questions on single document images and asks for answers that appear in the page text. That tests reading. A scanned document revision benchmark tests judgment across files: the model has to compare revision letters, signature blocks, stamps and dates, notice that one copy is a draft and another is superseded, and sometimes admit that none of the candidates is approved.

This guide gives the definitions a labeler needs, the regions of a scan that carry version evidence, a field-level specification for the test set, a distractor catalog, a scoring rule, an illustrative worked item and the cases where a single right answer does not exist.

Final, latest and approved are three different labels

Decide what the gold label means before anyone labels a file, because the newest scan is often not the approved one. A later scan can be a marked-up working copy, a rescan of an older revision pulled from a binder, or a draft circulated for comment after the approved version was issued. Define four states and apply them per file: draft (not approved), approved (an authorized person signed or stamped it), effective (approved and in force on the question date), and superseded (approved once, replaced by a later approved revision).

Approval needs evidence of who approved, when, and in what capacity. In FDA-regulated electronic records, 21 CFR 11.50 requires signed records to show the signer's printed name, the date and time of signing, and the meaning of the signature, such as review, approval, responsibility or authorship. Scanned paper rarely meets that standard, but the same three elements make a good labeling rule for any industry: a signature without a stated meaning or date is weaker evidence than one with both.

Write the question date into every item. The same set of scans has different correct answers on different dates, and a test that omits the date rewards models that simply pick the highest revision letter.

Final, latest and approved are three different labels
LabelDefinition used in the test setTypical evidence on the scanCommon false signal
DraftNot approved by an authorized personDraft watermark, blank signature block, for comment stampA clean copy with no watermark that was never signed
ApprovedSigned or stamped by an authorized approver with a dateCompleted signature block, approval stamp, approval date in a revision tableA reviewer initial that records review, not approval
EffectiveApproved and in force on the question dateEffective date field, release stamp, distribution record in another fileAn approval date that precedes a later approved revision
SupersededApproved once, replaced by a later approved revisionSuperseded or obsolete stamp, later revision in the set, change noticeAn old copy with no stamp because nobody marked it

Which regions of a scan carry the version evidence?

Version evidence clusters in a few places, and the test set should record each one as a page number plus a bounding box so that grounding can be scored. On engineering drawings it sits in the title block and revision table; on procedures and specifications it sits in the header, footer and approval page; on contracts and forms it sits in the signature block and any handwritten amendment in the margin.

At a glance

Which regions of a scan carry the version evidence

Record OCR confidence for the words inside each evidence region, because a model that misreads a revision letter fails for a reading reason, not a judgment reason, and the error analysis should be able to tell them apart. The ALTO XML schema version 4.4 defines a word confidence attribute between 0 (unsure) and 1 (sure), and the hOCR specification defines an engine-specific x_wconf property where higher values express higher confidence. Store whichever your OCR engine emits, with the engine name, since the scales are not comparable across engines.

Capture scan resolution with each file. Tesseract's documentation says the engine works best on images of at least 300 dpi, so a low-resolution fax copy is a fair test of robustness but should be tagged so results can be split by image quality.

  • Title block or document header: document number, title, revision letter or number, date.
  • Revision table or change history: each revision, its description, date and approver.
  • Signature or approval block: signer name, role, date, and the stated meaning of the signature.
  • Stamps and watermarks: draft, approved, released, superseded, controlled copy, uncontrolled copy.
  • Handwritten marks: redlines, margin amendments, initials, struck-through values.
  • Footer and page count: page x of y, which exposes missing approval pages.

The document version selection test set, field by field

Each test item is one record with the fields below. The specification is deliberately strict about provenance and cross-file links, because the hardest items are the ones where the scan alone cannot settle the answer and the labeler relied on a change order, a document control log or an email approval. Federal digitization rules point the same way: 36 CFR 1236.54 requires unique file names and record IDs for each image file and relation metadata for records made of multiple parts, which is the minimum a test set needs to keep candidates and their evidence linked.

Assemble candidate sets from whole document histories rather than from random pairs. Pull every scan that shares a document number, add copies found under a different file name by matching titles and content, and only then decide which belong in the item. Split train and test by document family, not by item, so that no revision history appears on both sides; otherwise a model can memorize which copy of a given procedure was approved instead of learning to read approval evidence.

Freeze each item once adjudicated and give the set a version number and date. If a later control log changes a gold answer, issue a new set version and keep the old one, so results stay comparable over time.

The document version selection test set, field by field
FieldWhat it holdsValidation checkWhy it matters
item_id and question_dateStable item ID and the date the question asks aboutUnique ID; date present and in ISO formatThe same candidates have different answers on different dates
candidate_filesTwo or more scan IDs, each with page count, dpi and source systemEvery ID resolves to a file; dpi recordedDefines the choice set and supports quality splits
gold_selectionOne candidate ID, or none_qualifiesExactly one value; none_qualifies allowedTests abstention as well as selection
per_file_stateDraft, approved, effective or superseded for every candidateOne state per candidate; one effective file at most unless the item is a declared exceptionLets you score near misses and diagnose errors
evidence_regionsPage and bounding box for each region the labeler usedBox lies within page bounds; at least one region for the gold fileScores grounding, not just the final pick
evidence_typeRevision table, signature block, stamp, watermark, handwritten mark or external recordValue from a closed listShows which evidence types models miss
ocr_layerEngine name, version and word confidence for each evidence regionEngine and version present; confidence on the engine's own scaleSeparates misreading from misjudging
cross_file_linksIDs of change orders, control logs or approval emails the labeler consultedEach ID resolves; empty only when the scan alone decidesMarks items that need context beyond the scan
distractor_tagsWhich distractor families each wrong candidate representsValues from the distractor catalogLets you report accuracy by difficulty
label_provenanceLabeler ID, adjudicator ID, date and agreement statusTwo independent labels; disagreements adjudicatedMakes the gold answer auditable

Which distractors make the benchmark hard enough to matter?

A test set built only from clean pairs of an old and a new revision is too easy, because a model can pick the higher revision letter and score well. Real document stores contain copies that look final and are not, and the benchmark should sample each family below in proportions the team declares in advance. Tag every wrong candidate with its family so results can be broken down.

Which distractors make the benchmark hard enough to matter?
Distractor familyWhat the wrong candidate looks likeWhat the model must noticeWhere it comes from
Unsigned finalIdentical content to the approved file but the signature block is blankThe missing approval, not the contentCopies printed before signing
Superseded approvedFully signed and stamped, but a later approved revision existsThe later revision in the set or a superseded stampOld binders and shared drives
Marked-up working copyNewest scan, covered in redlinesHandwritten changes mean it is a proposalReview cycles scanned for the record
Rescan of an older revisionRecent scan date, older revision letterScan date is not revision dateBulk back-file scanning projects
Wrong approverSigned, but by a role that cannot approve this document typeSigner role against the approval ruleDelegation errors and stand-in signatures
Amended finalApproved file with a later handwritten margin change and initialsWhether the amendment was itself approvedField changes recorded on paper
Missing approval pageContent pages present, approval page absentPage x of y and the gapPartial scans and split PDFs
No qualifying candidateEvery candidate is a draft or supersededThat the right answer is to abstainRequests where the approved copy was never scanned

How should a revision benchmark score answers?

Score selection, grounding and abstention separately and report each, because a single accuracy number hides the failure that matters most in production: confidently choosing a superseded copy. Selection is correct when the predicted file equals the gold selection. Grounding is correct when the model's cited region overlaps a gold evidence region on the gold file by at least the threshold your team declares before running the test. Abstention is scored on none_qualifies items and on items where the model declines.

Report selection accuracy split by distractor family, by image quality and by whether the item needed cross-file links. Weight errors by consequence if the use case warrants it: picking a superseded approved revision is usually worse than abstaining, and the scoring rule should say so in writing before any model is run.

Do not reuse reading metrics as the headline. DocVQA scores extracted answer strings against page text with a normalized string similarity, which suits extraction. A version selection item has a categorical answer, so exact match on the file ID plus a grounding check is the honest measure.

  • Selection: predicted file ID equals gold_selection.
  • Grounding: cited region overlaps a gold evidence region on the gold file above the declared threshold.
  • Abstention: correct when the model declines on a none_qualifies item; counted as an error when it declines on an item with a valid answer.
  • Severity: superseded or draft picks reported apart from other errors.
  • Splits: distractor family, dpi band, OCR engine, cross-file link required or not.

Worked example (illustrative): revision C, revision D and an unsigned rescan

This item is invented for illustration and does not come from any company's records. A manufacturer's inspection procedure exists as three scans. File 1 is revision C, signed by the quality manager with the meaning approved and a date. File 2 is revision D, identical in layout, with revision D in the title block and a completed revision table entry, but the approval signature block is blank. File 3 was scanned most recently; it is revision C again, pulled from a work-cell binder, with a handwritten margin change to one inspection interval and initials but no date.

The question asks which file is the approved version in effect on a date after all three scans exist. The gold answer is File 1. File 2 is newer but unapproved; the labeler confirmed through the document control log, recorded as a cross-file link, that revision D was still in review on the question date. File 3 is an amended copy of revision C whose margin change was never approved. Evidence regions are the signature block on page 1 of File 1, the blank signature block on File 2, and the margin note on page 3 of File 3.

A model that answers File 2 has fallen for the newest revision letter. A model that answers File 3 has trusted the most recent scan date. A model that answers File 1 but cites only the title block gets selection right and grounding wrong, which is still useful to know.

The exception: had the control log shown that revision D was approved electronically and the paper copy simply never signed, the gold answer would change to File 2 and the item would carry evidence_type external record. That is why the cross_file_links field exists.

When a single right answer does not exist

Some document families have more than one valid approved version at once, and forcing a single gold answer into those items teaches the model something false. Site-specific variants of the same procedure, documents approved per customer or per jurisdiction, and controlled copies issued to different departments can all be simultaneously effective. Either narrow the question so one answer is correct, for example by naming the site, or exclude the family from the test set.

Exclude items where the approval lived only in a system you cannot show the labeler, such as an e-signature platform that was not exported, unless the item is explicitly built to test abstention. Send to a records management specialist any item where the labelers disagree after adjudication, where the approval authority depends on an internal delegation policy, or where the document is regulated and the approval rule comes from a regulation rather than company practice.

The limits of the method are worth stating on the results page: a benchmark built from one industry's documents measures that industry's conventions, revision schemes vary between letters, numbers and dates, and a model that does well on title blocks may still fail on forms where approval is a stamp.

Sourcing scanned revision histories with their approval trail

The most useful raw material is a document control archive where every revision was scanned and the control log, change orders and approval records survive alongside the images. Back-file scanning projects, quality system archives and engineering document stores often hold exactly this, including the messy copies that make good distractors. Having access to those scans is not the same as having the right to license them: customer drawings, supplier documents and contracts with confidentiality clauses each need their own review, and removing names does not create a licensing right.

AI teams can commission such archives through SourceX, which seeks them out at U.S. businesses only after a request specifies scope, volume, format, timing and licensing terms. The supplying company approves scope and terms and signs before any records are delivered, and SourceX handles rights review, preparation and delivery between the two.

Frequently asked questions

What should a document version selection test set include?

Each item needs a question date, two or more candidate scans, one gold selection or an explicit none_qualifies answer, a state label for every candidate, the page regions that prove approval, and links to any outside records the labeler used. OCR confidence and scan resolution per evidence region let you separate reading errors from judgment errors.

Which page region holds the evidence that a scan is the approved version?

Usually the signature or approval block, the revision table and the title block, with stamps and watermarks as supporting evidence. Handwritten margin changes and missing approval pages are the regions models most often overlook, so record them as evidence regions when they decide an item.

Why not just pick the scan with the highest revision letter?

Because the highest revision may still be in review, and the most recent scan may be a rescan of an older revision. A benchmark that rewards the highest letter measures sorting, not approval recognition, which is why distractors such as unsigned finals and rescans belong in the set.

How many labelers does each item need?

Two independent labelers with adjudication of disagreements is a sound default, and the adjudicator's decision should be recorded in the item's provenance. Items that still divide reviewers after adjudication are better excluded or sent to a records specialist than kept with a contested gold answer.

Should the model be allowed to say none of the files is approved?

Yes, and it should be scored on that ability. Real requests often arrive with only drafts and superseded copies, and a model that always picks something will confidently return the wrong document in exactly those cases.

Sources

  • Signed electronic records must show the signer's printed name, the date and time of signing, and the meaning of the signature, such as review, approval, responsibility or authorship. Source
  • Federal digitization metadata rules require a unique file name and record ID for each image file and relation elements for multi-part records. Source
  • ALTO 4.4 word confidence (WC) is a value between 0 (unsure) and 1 (sure) for the OCR of a string. Source
  • The hOCR x_wconf property is an OCR-engine-specific confidence for the contained substring, where higher values express higher confidence. Source
  • Tesseract works best on images with a resolution of at least 300 dpi. Source
  • DocVQA is a visual question answering dataset defined on single document images, with answers taken from the document text and scored by normalized string similarity. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies