Privacy and preparation
Multimodal dataset accessibility metadata: alt text, OCR or originals?
By SourceX Editorial · Updated
9 sections
Short answer
Buyers should receive the original image or file, region-anchored OCR with the engine's native confidence, and alt text or long descriptions only where the source or a stated guideline produced them, labelled by origin. Tie every layer to the same file hash, page and region box in one declared coordinate system, so any answer can be traced back to the pixels it depends on.
Document stack
Multimodal dataset
Key takeaways
- Originals, OCR and alt text are separate layers with different authors and reliability; none replaces the others.
- Region-anchored OCR with native confidence is the default layer for document AI; plain text dumps lose the evidence link.
- hOCR, ALTO, Textract and IIIF use different units and confidence scales, so declare one coordinate system and keep native values.
- Source alt text is a useful label only when it describes purpose; placeholders and filenames should be dropped and generated text labelled.
- A redaction is incomplete until every text layer and metadata field is re-anchored or cleaned.
What should buyers receive alongside images: OCR, alt text or original files?
Buyers should receive the original file, region-anchored OCR with the engine's own confidence, and alt text or long descriptions only where they exist in the source or were written under a stated guideline, with every layer tied to the same file hash, page and region. None of the three replaces the others. The original holds the evidence, OCR makes the text in it addressable, and alt text records what a human thought the image was for.
The mistake most packages make is shipping one layer and losing the links. A folder of page images with a separate text dump cannot tell a model which words came from the table and which from the stamp in the corner. A set of alt text strings without the images teaches a model to describe images it never saw.
This page compares the common delivery packages, gives the per-image fields that keep the layers linked, and explains where each layer breaks.
Alt text, OCR and the original file answer different questions
Alt text says what an image means to a reader, OCR says what characters appear and where, and the original file is the image itself. They differ in who made them, how reliable they are, and what a model can learn from them.
Alt text is an accessibility artifact. The W3C's guidance on WCAG success criterion 1.1.1 says a text alternative should serve the equivalent purpose of the non-text content, and its images tutorial says complex images such as charts, diagrams and maps need a two-part alternative: a short description plus a long description of the essential information. That makes alt text interpretive by design. OCR, by contrast, is a transcription produced by an engine, with geometry and a confidence value per word.
| Layer | Produced by | Useful for training | What it cannot do |
|---|---|---|---|
| Original file | The business system that created or received it | Visual grounding, layout, handwriting, stamps, photos, ground truth for every other layer | Say which part matters or what the text reads, without further work |
| OCR with geometry | An OCR engine, with version and settings | Text-region alignment, document extraction, reading order, table structure | Interpret a chart or photo, or flag its own misreads beyond a confidence score |
| Alt text or long description | The document author, an annotator or a model | Image-to-text description, chart summarization, accessibility tasks | Prove what pixels contain; it reflects the writer's purpose and skill |
| Embedded text layer | The authoring tool when the PDF was born digital | Exact text for born-digital pages with no OCR error | Cover scanned pages, image regions or text drawn as graphics |
The multimodal delivery package comparison
Use the comparison below to pick a package before a sample is cut. The full linked package costs the most preparation, and it is the only one that supports grounded evaluation, where a grader checks that an answer came from the right region of the right page.
| Package | What it contains | Good for | Breaks when | Verdict |
|---|---|---|---|---|
| Images only | Page or photo images with file-level metadata | Self-supervised visual pretraining | Any task needs text, labels or regions | Rarely enough for business documents |
| Images plus plain OCR text | One text file per page, no coordinates or confidence | Bulk text pretraining | A task asks where a value came from or how sure the reading is | Cheap, but loses the evidence link |
| Images plus region-anchored OCR | Word and line boxes with confidence in hOCR, ALTO or a JSON schema | Extraction, layout and grounded question answering | Coordinates use an undeclared system, or pages were re-rendered after OCR | The default for document AI |
| Images plus alt text and long descriptions | Text alternatives tied to image regions, with origin labels | Chart and figure description, accessibility models | Alt text is missing, placeholder or model-written without a label | Add when figures carry meaning |
| Full linked package | Originals, embedded text, region OCR, alt text, table structure and cross-file links under one manifest | Grounded evaluation, multi-document reasoning, audit | The manifest and hashes are not versioned with the files | Best for evaluation sets and high-value training |
| Rendered or redacted images plus layers | Derived images with personal data removed, layers re-anchored to the rendering | Privacy-sensitive records | Layers still point to original coordinates or text | Use when originals cannot leave the source |
How should OCR and alt text point to the exact region of the page?
Every region needs a box in a declared coordinate system, the page dimensions that system refers to, and the confidence on its native scale. Common OCR formats disagree on all three, so a package that mixes engines without saying which convention each row uses will misplace boxes and compare confidences that do not mean the same thing.
The hOCR specification defines a bounding box as left, top, right and bottom in pixels measured from the top-left corner of the image, and its word confidence property is engine-specific. ALTO, maintained by the Library of Congress, measures in tenths of a millimetre by default unless the file declares another unit, and records word confidence from 0 to 1. Amazon Textract returns boxes as ratios of page width and height between 0 and 1, along with polygons, and reports confidence as a percentage. Tesseract can write hOCR, TSV with per-word boxes and confidence, and a searchable PDF that pairs the image with a text layer.
For pointing at a region from another record, the IIIF Image API offers a compact convention: a region written as x,y,w,h in pixels from the upper-left pixel, or as pct:x,y,w,h in percentages of the full image. Whatever convention you choose, publish it once in the manifest and convert every engine's output into it, keeping the native values alongside.
| Format | Box convention | Units | Confidence scale |
|---|---|---|---|
| hOCR | x0 y0 x1 y1, left top right bottom, origin top-left | Pixels of the document image | x_wconf, engine-specific number |
| ALTO | Horizontal and vertical position with width and height | Tenths of a millimetre by default, or the declared unit | 0 (unsure) to 1 (sure) for words |
| Amazon Textract | Left, Top, Width, Height plus a polygon | Ratios of page width and height, 0 to 1 | Percentage |
| Tesseract TSV | left, top, width, height per word | Pixels | conf column, with -1 on non-word rows |
| IIIF region | x,y,w,h or pct:x,y,w,h | Pixels or percent of full image | Not applicable; addressing only |
Which fields should every image record carry?
Carry enough to rebuild the link from any answer back to the pixels it depends on. The list below is the per-image record a buyer can ask for by name; the first five fields identify the evidence, the rest describe each layer and how it was made.
- source_file_sha256 and source_file_type: the hash and type of the original, so every layer can be checked against the exact bytes.
- page_number, page_width, page_height and resolution: the frame every coordinate refers to.
- rendering_id: set when the delivered image is a rendering or redaction rather than the original, with the tool and settings used.
- region_id, region_box and region_type: one row per region, typed as body text, table, figure, photo, stamp, signature, handwriting or form field.
- coordinate_system: one value for the whole package, with native coordinates kept in a separate column when converted.
- ocr_engine, ocr_version, ocr_text and ocr_confidence_native: confidence stays on its original scale, never silently rescaled.
- alt_text, long_description and text_alternative_origin: origin is source-authored, annotator-written under a named guideline, or model-generated with the model named.
- table_headers and footnote_links: header cells linked to the data cells they govern, and footnote markers linked to the note text.
- cross_file_links: identifiers of the related order, ticket, invoice or email that the image is evidence for.
- layer_version: incremented whenever OCR, alt text or region boxes are regenerated.
When is source alt text worth keeping as a label?
Keep source alt text when it describes the image's purpose in a sentence a person could use, and tag it with its origin; drop it from training targets when it is a placeholder. Business documents are full of alt text that fails that test, because authoring tools fill it in automatically or authors skip it.
The W3C lists using placeholder text such as image or picture, or a filename, as a failure of the non-text content criterion. Those strings are worse than nothing as training targets: a model learns that the right description of a wiring diagram is diagram_final_v2.png. Decorative images are the opposite case and should carry empty alt text, which is itself a useful label that the image adds no information.
Model-written alt text needs the most care. It is a derivative of another model's output, so it should be labelled as generated, and it should not be used as a ground-truth label for evaluating a model of the same family.
Worked example (illustrative): an inspection report where the answer sits in one photo
This example is invented to show the structure. A field-service inspection report is a four-page PDF: two pages of born-digital text, one page with a parts table, and one page with three photos. The evaluation question asks for the serial number of the replaced pump.
In the full linked package, the serial appears only in photo two, on the equipment nameplate. Region-anchored OCR finds it with a word confidence of 0.81 on ALTO's 0 to 1 scale, and the region row links it to the work order and the parts table line for the replacement. The source alt text for the same photo reads photo of equipment, which is accurate and useless. A grader can verify an answer by checking that it cites photo two's region and matches the OCR string, then a person confirms the digits against the pixels.
The exception: in an illustrative variant of the same report, the nameplate in photo two is partly covered by a technician's hand and OCR confidence falls below the package's review threshold. That region is flagged for human transcription instead of being dropped, because removing hard cases is how evaluation sets end up easier than real work.
With a plain OCR text file instead, the serial would sit in a page of text with no indication that it came from a photo, and the question of whether the model read the photo or guessed from the parts table could not be answered.
When should a rendering or redaction replace the original?
Replace the original when it holds personal or confidential information that the licensing terms or privacy review do not permit, and only if every layer is re-anchored to the rendering. A redaction that blacks out a name but leaves the OCR text, the embedded text layer or the image metadata untouched has redacted nothing.
Stop and request specialist review in three situations: the images include health, financial or identity documents; handwriting or signatures are central to the task; or the alt text or OCR will be used as ground truth for an evaluation that decides a purchase. Each needs a records expert or a document AI engineer to confirm the layer is reliable enough for that use.
- Re-run OCR on the rendered image, or delete text that falls inside redacted regions from every layer.
- Strip or rewrite embedded metadata such as author fields and EXIF location data.
- Record rendering_id, the redaction method and the regions affected, so a buyer knows which pixels are synthetic.
- Check that region boxes still align after any change in resolution or page size.
Requesting document images with every layer linked
Name the package in the request. A buyer who asks for documents gets whatever is easiest to export; a buyer who asks for originals with region-anchored OCR, origin-labelled alt text and a manifest of hashes gets something that can be audited.
Image collections with linked layers can be requested from SourceX, which sources them from U.S. businesses when asked rather than keeping a stock, with scope, volume, format, timing and licensing needs set out in the request. Rendering and re-anchoring choices are made during de-identification, which SourceX performs through approved processing routes, and very large collections remain in the seller's storage or move on encrypted drives.
Frequently asked questions
Is OCR text a substitute for alt text in a training dataset?
No. OCR transcribes the characters in an image and says where they are, while alt text describes what the image is for. A chart can have perfect OCR of its axis labels and still need a long description of the trend it shows.
Should a dataset include the original files or just extracted text?
Include the originals whenever licensing and privacy review allow it. They are the ground truth every other layer is checked against, and they carry layout, photos, stamps and handwriting that extracted text loses.
How do I compare OCR confidence from different engines?
Keep each engine's native value and its scale, because they are not equivalent. hOCR word confidence is engine-specific, ALTO uses 0 to 1 and Amazon Textract reports a percentage. Calibrate thresholds per engine on a sample with human-checked transcriptions.
What coordinate system should region boxes use?
Any one system works if it is declared once in the manifest and every row is converted into it. Pixel boxes from the top-left corner with stated page dimensions are the most common, and percentage boxes survive resizing.
Can model-generated alt text be used as labels?
It can be used for training if it is labelled as generated and the generating model is named. Avoid it as ground truth for evaluation, especially of a model from the same family, because the test would reward agreement with another model rather than accuracy.
Sources
- W3C's Understanding document for WCAG success criterion 1.1.1 says text alternatives should serve the equivalent purpose, with a short alternative plus a long description where a short one cannot, as for charts and diagrams. Source
- The W3C WAI images tutorial says complex images such as graphs, charts, diagrams and maps need a two-part text alternative: a short description and a long description of the essential information. Source
- W3C technique F30 describes using placeholder text such as spacer, image or picture, or filenames, as text alternatives as a failure of success criterion 1.1.1. Source
- The hOCR 1.2 specification defines bbox as x0 y0 x1 y1 in pixels from the top-left corner of the document image and x_wconf as an OCR-engine-specific confidence. Source
- The Library of Congress ALTO description says measurements default to tenths of a millimetre unless another unit is declared, and records word confidence from 0 (unsure) to 1 (sure). Source
- Amazon Textract documentation says bounding box values are ratios of the page width and height between 0 and 1, polygons are also returned, and confidence is reported as a percentage. Source
- Tesseract's command-line documentation shows hOCR output with word bounding boxes and x_wconf, TSV output with per-word boxes and a conf column, and searchable PDF output with a separate text layer. Source
- The IIIF Image API 3.0 specifies image regions as x,y,w,h in pixels from the upper-left pixel or as pct:x,y,w,h percentages of the full image. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.