Skip to content

Systems and records

Document visual provenance: pointing each answer to its page evidence

By SourceX Editorial · Updated

10 sections

Short answer

A document AI answer should cite every page region it relied on, not just one box around the answer: the file version, page number, region coordinates in a declared unit, the exact text quoted, and the role each region played, such as answer value, column header, footnote or superseding notice. Record OCR confidence and cross-file links too, so a reviewer can reopen the page and check the answer in seconds.

Provenance trail

Document visual provenance

This page covers document visual provenance: pointing each answer to its page evidence.Illustrative

Key takeaways

  • Cite every region an answer depends on, including headers, units, footnotes and version blocks, not only the answer value.
  • Give each evidence region a role so models and evaluators can check that qualifying context was used.
  • Declare one coordinate unit and origin per dataset and store page size with each region.
  • Anchor regions twice, with geometry and an exact quote, and pin both to a hash of the file version.
  • Write explicit rules for computed, absent, chart, handwritten and redacted evidence before annotation starts.

How should a document AI answer point to its supporting page evidence?

It should point to a list of evidence regions, each tied to an exact file version and page, with coordinates a viewer can draw and a quoted string a reviewer can match. That is what document visual provenance means in practice: anyone holding the answer can reopen the same page, see the highlighted regions and confirm that they say what the answer claims.

The pointer has to cover everything the answer depends on. A value read from a table cell is only correct under its column header, its unit and any footnote attached to it, and only for the document revision that was in force. If the citation shows the cell alone, the reviewer has to rediscover the rest, and a model trained on that citation learns that the rest does not matter.

The same pointer serves three audiences. A reviewer uses it to accept or reject an answer quickly, a training pipeline uses it as grounding supervision, and an evaluator uses it to score whether a model looked in the right places. Designing the record once for all three avoids relabeling the same pages later.

Why one answer box is not enough provenance

Because the answer region is rarely the whole reason the answer is right. Single-box labels teach a model where the string was, not why it applies, and they break as soon as a document is re-rendered, re-scanned or replaced by a newer version.

Business documents make this worse than benchmark pages. Spec sheets carry unit rows and derating footnotes, insurance schedules carry endorsements on later pages, purchase terms refer to attachments, and many documents are superseded by a notice stored in a different file. An answer box that ignores those regions produces confident answers that are wrong for the case at hand, and an evaluation built on it rewards them.

Region-level evidence fixes both problems. It records the supporting regions explicitly, gives each one a role, and stores enough anchoring information that the regions can be found again after the file is processed differently.

What should the region-level evidence annotation specification include?

It should define one evidence record per region, grouped under the answer it supports, with fields precise enough that two annotators produce the same record. The specification below is written to be used as is; adjust the controlled vocabularies to your document types.

Two design choices matter more than any single field. First, the record is per region, not per answer, so a model can be trained or scored on whether it found each supporting region, and an answer with a missing qualifier shows up as partially grounded rather than simply right or wrong. Second, the human-read quote and the machine-read OCR text are stored separately. The quote is the ground truth; the OCR text shows what a model working from the text layer would have seen, and the difference between them is often the most useful error signal in the dataset.

What should the region-level evidence annotation specification include?
FieldDefinitionAnnotation ruleIllustrative value
answer_idThe question and answer this region supportsOne answer can have many regions; each region belongs to one answerq-0412
file_id and file_hashStable ID plus a content hash of the exact fileHash the original bytes, not a converted copyspec-77, sha256 of the PDF
pagePhysical page number, first page is 1Use physical pages, never printed folio numbers3
region and unitPolygon or box plus the unit and origin usedDeclare unit and origin once per dataset and never mixx 1.20 y 4.85 w 2.10 h 0.30, inches, top-left
page_sizeWidth and height of the page in the same unitRequired so coordinates can be normalized later8.5 by 11 inches
quoteExact text inside the region, as read by a personCopy characters exactly, including symbols and unitsmax 70
roleWhy the answer depends on this regionPick one value from the evidence role listanswer_value
ocr_text and ocr_confidenceMachine-read text and the engine's confidenceKeep both even when the quote differs; flag low confidencemax 7O, 0.61
linksOther regions or files the region must be read withPoint to region IDs, not to whole documentsr-3, ecn-12 page 1
annotator and reviewWho drew it and who checked itSecond check on a sample and on every low-confidence regionann-04, rev-02

Which evidence roles does a single answer depend on?

Most answers depend on three to five regions with different jobs. Naming those jobs is what lets a model, or an evaluator, check that an answer used the qualifying context and not only the value.

Keep the vocabulary short. Annotators agree well on a handful of roles defined by what the reader would lose without the region, and agree poorly on fine-grained layout categories. If a region could take two roles, the rule is to choose the one that changes the answer: a footnote that only repeats the header is context, while a footnote that limits the value is a qualifier. Measure agreement on the role field separately from agreement on box placement, because they fail for different reasons and need different guideline fixes.

Which evidence roles does a single answer depend on?
RoleWhat it isTypical locationRequired when
answer_valueThe region that contains the value or statement given as the answerTable cell, form field, sentenceAlways
header_or_labelRow or column header, field label or unit that gives the value its meaningTable header row, label to the left of a fieldThe value sits in a table or form
qualifierFootnote, condition, exception or note that limits the valueFootnote, asterisk note, fine printAny marker links the value to a note
version_contextRevision, effective date or status block that says which version appliesTitle block, header, footerDocument has revisions or dates
superseding_referenceA region in another file that replaces or amends this oneChange notice, amendment, endorsementA later document changes the answer
searched_without_matchRegions checked to support an answer that something is absentRelevant section or tableThe answer says the document does not state it

Which coordinate convention should the regions use?

Pick one convention for the dataset, store page size with every region, and convert at export. The common extraction services and formats each define their own, and mixing them is the most frequent cause of boxes that land in the wrong place.

A practical default for a mixed corpus is to annotate in normalized ratios of page width and height, with the page size stored alongside, and to generate vendor-specific or viewer-specific coordinates in an export step. Ratios do not change when the same page is rendered at a different resolution, which is the usual reason boxes drift between annotation tools and training pipelines. Whatever you choose, write the unit, origin and axis direction into the dataset card and test the export by drawing a sample of boxes back onto rendered pages.

Azure AI Document Intelligence returns bounding regions with a 1-based page number and a polygon measured from the top-left of the page, in inches for PDFs and pixels for images, and links elements to the concatenated text through spans of offset and length. Amazon Textract reports bounding boxes as ratios of page width and height between 0 and 1. The hOCR format writes bbox x0 y0 x1 y1 relative to the binarized page image and offers x_wconf for engine-specific word confidence. For links that open a viewer at the right place, RFC 8118 defines PDF fragment parameters such as page, with the first page numbered 1, and the W3C Media Fragments Recommendation defines xywh rectangles in pixels or percent from the top-left of an image.

Which coordinate convention should the regions use?
ConventionUnitOrigin and page numberingWhen to use it
Azure Document Intelligence polygonsInches for PDF, pixels for imagesTop-left of page; pageNumber starts at 1When the pipeline already uses Azure layout output
Amazon Textract geometryRatio of page width and height, 0 to 1Top-left of pageWhen pages are rendered at different resolutions
hOCR bboxPixels of the binarized page imageTop-left of that imageOpen-source OCR pipelines such as Tesseract output
PDF fragment link per RFC 8118Page selector for a viewerFirst page is 1Opening the cited page in a browser viewer
Media Fragments xywhPixels or percent of the imageTop-left of the imageHighlighting a region on a page image

How to anchor regions so they survive re-rendering and re-OCR

Store two anchors for every region: the geometry and the exact quote with a little context on each side. If either one changes after reprocessing, the other lets you find the region again and tells you that something moved.

The W3C Web Annotation Data Model already defines this pattern. A FragmentSelector can carry a region such as an xywh rectangle, and a TextQuoteSelector records the exact text with an optional prefix and suffix, so a quote that appears twice on a page can still be told apart. Combining the two, with the region refined by the quote, gives an evidence pointer that survives a new OCR pass better than either alone.

Pin everything to a file hash. If a supplier re-exports the same document with different fonts or a new scan, the hash changes and the dataset should treat it as a new file whose regions need checking, not silently reuse old coordinates. Keep the page size alongside each region so normalized and absolute coordinates can be converted without reopening the file.

A simple acceptance check follows from the two anchors. After any reprocessing, recompute each region's text from the new OCR output and compare it with the stored quote. Regions that still match can be carried over; regions whose text changed are re-reviewed; regions whose quote can no longer be found on the page are flagged as broken evidence and the answer is held back until someone checks it.

Worked example: a spec sheet value with a footnote and a superseding change notice

Consider the question: what is the maximum operating temperature for part 77 under the current revision? Every value in this example is illustrative. The answer value is a cell reading 70 in the Max column of an environmental table on page 3 of revision C of the part's spec sheet.

Four more regions are needed to make that answer checkable. The column header on page 3 gives the unit, degrees Celsius. A footnote marker on the cell points to a note at the bottom of the page saying the limit is derated above an altitude threshold. The title block on page 1 shows the revision letter and effective date. And a separate engineering change notice, issued after revision C, amends the table for one variant of the part.

The evidence record therefore holds six regions: answer_value, header_or_label, qualifier, version_context, the superseding_reference on the change notice and, because the change notice applies only to one variant, a second qualifier on the notice naming that variant. The model's answer becomes 70 degrees Celsius for the standard variant, derated at altitude, with the amended value for the listed variant. An evaluator can check each part of that sentence against a region rather than against a judgment call.

The example also shows why OCR confidence belongs in the record. The scanned change notice reads the variant code with low confidence, so the specification routes that region to a second annotator before the record is accepted.

Computed answers, missing answers, charts and handwriting

Each of these breaks the one-value, one-region assumption, so the specification needs an explicit rule for them rather than leaving annotators to improvise.

At a glance

Computed answers, missing answers, charts and handwriting

Write these rules into the guideline before the first batch, with one example page for each, because annotators who meet them without a rule tend to draw a single box around the nearest plausible text and move on.

  • Computed answers: when the answer is a total, difference or count, record every operand region as answer_value and add a short note naming the operation, so the calculation can be repeated.
  • Answers that something is absent: record the sections or tables the annotator searched under searched_without_match, and mark the answer as an abstention, not a value.
  • Charts and diagrams: when a value is read from a plotted bar or line, the region is the mark on the chart plus its axis labels, and the quote field states the value as read, flagged as visually estimated.
  • Handwriting and stamps: keep the image region as the evidence and treat transcribed text as an annotation with its own confidence, because OCR text alone is not the evidence.
  • Redacted regions: never point evidence at a redaction box; if the redaction removed the evidence, the question is unanswerable from the delivered file and should be marked that way.

Signals that a document set is not ready for region-level evidence

Stop and narrow the scope when the files cannot be pinned to versions, when the same document exists in several renderings with no way to tell which one was used, or when OCR confidence is low across most pages. Region-level evidence on unstable inputs produces labels that look precise and are not.

Bring in a records specialist when supersession is unclear, for example when amendments, endorsements or change notices are stored outside the document system or referenced only by number. Bring in a privacy reviewer when the evidence regions contain personal details such as names, signatures or account numbers, because the coordinates and quotes travel with the dataset and can reveal what de-identification removed from the text layer. If the answer depends on linked records that are not in the delivery, record the link and mark the answer partial rather than drawing evidence that does not exist.

Sourcing documents together with the records that explain them

Region-level evidence is most valuable when the documents arrive with the records that explain them: the revision history from a document control system, the change notices and amendments that supersede earlier versions, and the transaction or case records that show how a value was actually applied. Those links are what let an annotator fill the version_context and superseding_reference roles instead of guessing.

Buyers who need document sets with their revision trail attached can ask SourceX to find U.S. businesses whose files fit the scope, volume, language, format, timing and licensing needs they give, for example multi-revision technical documents paired with their change notices. SourceX does not host very large collections; those remain in the supplying business's storage or travel on encrypted drives.

Frequently asked questions

What is document visual provenance?

It is the record that ties an answer extracted from a document to the exact file version, page and regions it came from, with coordinates and quoted text. It lets a person reopen the page and check the answer visually. It differs from data lineage, which tracks how a dataset moved between systems.

Should evidence coordinates be absolute or normalized?

Either works if the dataset declares its unit and origin and stores page size with each region. Normalized ratios survive different render resolutions, while absolute units match the PDF layout. Store page size so you can convert between them at export.

How many regions should an answer cite?

As many as the answer depends on, and no more. A plain field value may need two regions, the value and its label, while a table value with a footnote and a later amendment may need five or six. Unused context regions add noise to training and scoring.

Is a quoted text span enough without a bounding box?

Not for page-image models or human review. The quote proves what was said, but the region shows where, which matters when the same text appears twice or when layout carries meaning. Keep both, and use the quote to relocate the region after re-OCR.

What should happen when OCR confidence on an evidence region is low?

Keep the region, record the engine's confidence and route it to a second annotator who reads the image directly. If the text still cannot be read with certainty, mark the answer as uncertain or exclude it from evaluation.

Sources

  • The W3C Web Annotation Data Model defines a TextQuoteSelector with exact, prefix and suffix, a FragmentSelector that uses fragment syntax, and refinement of one selector by another. Source
  • Media Fragments URI 1.0 defines xywh rectangles in pixels or percent measured from the top-left corner of an image. Source
  • Azure AI Document Intelligence returns bounding regions with a 1-based page number and a polygon from the page's top-left, in inches for PDFs and pixels for images, plus spans of offset and length into the extracted content. Source
  • Amazon Textract bounding box values are ratios of the page width and height between 0 and 1. Source
  • RFC 8118 defines PDF fragment identifiers, including a page parameter where the first page has the value 1. Source
  • The hOCR specification defines bbox x0 y0 x1 y1 relative to the binarized document image and x_wconf as an OCR-engine-specific confidence. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies