Reading Order Data for Multi-Column, Form and Mixed-Layout Pages

What a reading-order label actually encodes

A reading-order label is a ranking over page elements, stored as an ordered list of element IDs or as successor links between them. The label sits on top of a layout layer: you first need boxes for paragraphs, headings, tables, figures and form fields, and only then can you say which comes after which. That is why region detection sets and reading-order sets are separate purchases even when they share images; the document layout analysis datasets page covers the detection layer, and this page covers the sequence layer.

Two granularities dominate, and they are not interchangeable:

• Region level. Each layout block gets a rank. This is what RAG chunkers and Markdown converters need, because they serialize blocks, not words. Disagreement between annotators is mostly about where floating elements go.
• Word or line level. Each OCR token or text line gets a rank. LayoutReader's ReadingBank (Microsoft, EMNLP 2021) works at the word level, which lets a model repair the line ordering of OCR engines directly. Word-level labels are far more expensive to produce by hand and are almost always derived from source files instead.

Many production schemas store both: a global region order plus an intra-region token order. If you already maintain a combined spec for layout, tables and fields, keep reading order inside it; the document annotation schema page shows how the layers fit together.

How sidebars, footnotes, captions and form fields get sequenced

Non-linear elements need written rules, because "the order a human reads" is ambiguous for anything outside the main text flow. Most serialization bugs on business documents come from these elements, not from plain two-column text.

A workable convention set, which you should adapt and then freeze in your guidelines:

• Multi-column body text. Column 1 top to bottom, then column 2. Spanning headings and full-width tables break the page into horizontal bands, and columns are read band by band. XY-cut style heuristics often fail when a full-width element sits mid-page.
• Sidebars and callout boxes. Either a separate thread (ordered after the main flow, or after the paragraph that references them) or excluded from the main sequence with a thread_id. Pick one and state it.
• Footnotes. After the main flow on that page, or linked to their anchor. For RAG, a link to the anchor is usually more useful than a position.
• Captions. Immediately after (or before) their figure or table, consistently. A caption orphaned in the main text is a common chunking error.
• Running headers, footers and page numbers. Usually excluded from the sequence. Tagged-PDF practice (ISO 32000, which PDF/UA builds on) typically marks such pagination content as artifacts rather than part of the logical structure tree.
• Forms. Key then value, field by field, following the visual grid (row-major for most US business forms). Check boxes belong to the option text they mark. FUNSD shows why links matter: it labels question and answer entities and the links between them rather than a single page sequence [3]. See key-value extraction labels and checkbox and selection mark data for those layers.
• Tables. Treat the table as one region in the page sequence and leave cell order to the table-structure layer (table structure recognition data).

Where a page has genuinely parallel content, such as bilingual columns or a comparison layout, a partial order (a graph with some unordered pairs) is more honest than a forced total order. Your metric must then tolerate ties.

Where reading order comes from without annotation

Born-digital source files often contain reading order already, which makes them cheaper and more consistent than hand labels. Scans and phone photos carry no order at all and need either annotation or projection from a matched source file.

ReadingBank is the clearest example of the born-digital route: its authors derived reading order automatically from the XML of Word documents and paired it with rendered page images at large scale. The trade-off is domain. Order inherited from Word files reflects Word-authored layouts, which differ from typeset financial statements, insurance forms or scanned packets.

For business data, the strongest pattern is a matched pair: the native file (DOCX, XLSX-exported PDF, tagged PDF) plus the scanned or printed version of the same document. You project the native order onto OCR tokens from the scan and get scan-condition training data with near-free labels. Scan noise still breaks the projection on faxes and photocopies, so budget a human review pass; degraded document images covers those capture conditions.

Public reading-order sets and their limits

Public sets are useful baselines, but each skews to a domain and carries its own license, so check both before you train on or ship with them. Treat any public figure as the authors' report, not as your expected production accuracy.

• ReadingBank (LayoutReader). Word-level order derived from Word XML; the authors report strong results on their own test split and improved OCR line ordering. Strong for word-level pre-training; weak on forms, scans and financial layouts. Read the release license before commercial use.
• OmniDocBench. A parsing benchmark across diverse PDF types that includes reading-order annotations and scores order with normalized edit distance [1]. It is an evaluation set, not a training corpus.
• DocLayNet. Large, human-annotated, and diverse across document categories, but built for region detection [2]. Use its boxes as the layout layer and add your own order labels.
• FUNSD. 199 noisy scanned forms with entity labels and question-answer links [3]. Too small to train order on, but a useful stress test for form serialization.

The common gap is business layouts: dense multi-column contracts, invoices with remit-to sidebars, ACORD and claim forms, and mixed packets. If your failures cluster there, public data will not close the gap; the Document AI datasets hub maps which task layers have usable public options.

Reading order evaluation metrics that catch real failures

Score reading order with a sequence measure over element order, and report it separately from text accuracy so OCR errors do not mask ordering errors. A single aggregate parse score hides exactly the bug you are trying to fix.

• Normalized edit distance over the element sequence. OmniDocBench uses this for reading order [1]. It is easy to explain and penalizes misplaced blocks, but treats a swapped pair of adjacent paragraphs and a sidebar moved across the page similarly.
• Kendall's tau over matched elements. Counts discordant pairs, so it rewards getting most pairwise relations right. Compute it only over elements matched between prediction and gold by box overlap.
• Page-level BLEU and average relative distance. Used in the LayoutReader evaluation for word-level order. BLEU suits long token sequences; relative distance measures how far each token lands from its gold position.
• Downstream checks. Count chunks that cross a column boundary, captions separated from their figure, and form values attached to the wrong key. These are what break retrieval.

Report metrics per layout stratum (single column, multi-column, form, mixed) and per capture condition. A model can post a high average while failing every two-column contract.

Request template for reading-order data

A precise request names the unit, the conventions and the evaluation, which lets a supplier tell quickly whether their documents fit. Use the template below as a starting point.

Illustrative example: invented to show structure; it does not describe an available dataset.

request: reading_order_labels
documents:
  types: [commercial contracts, vendor invoices, insurance applications]
  layouts: {multi_column: 40%, form: 35%, mixed_with_sidebars: 25%}
  capture: [born_digital_pdf, scanned_300dpi, fax]
  native_source_paired: preferred   # DOCX or tagged PDF for label projection
labels:
  layout_layer: region boxes, classes per our schema v2
  order_unit: region            # optional: token order within text regions
  format: jsonl                 # one page per line
  record:
    page_id: "doc_0141_p3"
    regions:
      - {id: r1, cls: heading,   bbox: [72, 64, 540, 92]}
      - {id: r2, cls: paragraph, bbox: [72, 100, 300, 410]}
      - {id: r3, cls: paragraph, bbox: [312, 100, 540, 410]}
      - {id: r4, cls: footnote,  bbox: [72, 700, 540, 730]}
    order: [r1, r2, r3]
    side_threads: [{thread: footnotes, order: [r4], anchor: r2}]
    excluded: [page_header, page_number]
conventions_doc: required (sidebars, captions, footnotes, form key-value)
qa:
  double_annotation_sample: 10%
  agreement_metric: Kendall tau on matched regions
evaluation: normalized edit distance + per-stratum breakdown
documentation: data card with source, annotation method, intended use

Ask for annotation methods and intended use to be documented in a data card, since order conventions are invisible in the labels themselves [4]. For double annotation and resolving disagreements over floating elements, see inter-annotator agreement metrics and annotation adjudication.

Sourcing reading-order labels on real business documents

The hard part is rarely the label format; it is getting real business layouts with the rights to use them for training. Contracts, invoices and forms carry personal and commercial details, so the documents need rights review and redaction before anyone annotates them.

SourceX sources operational datasets, including business documents, from US companies on request and manages the licensing process; categories are not inventory, and a request does not guarantee a match. Each dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Personal details such as names, emails, phones and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. If paired native files and scans would let you project order instead of hand-labeling, say so in your buyer request.

For adjacent layers, see PDF parsing training data for page-to-Markdown pairs and multi-page long document data for order across pages. The AI data hub lists every buyer guide.

Request reading order data for your layouts

Describe the documents, layouts and label unit you need, and SourceX looks for US businesses that hold matching data. Every release is approved by the supplying company, and nothing is contracted until a supplier agrees. Start a buyer request.