Short answer
Image-text pairs from procedures link a figure, photo or diagram to the step text it illustrates. Producing them takes layout parsing to find figures, captions and callouts, then linking each image to the correct step, not just the nearest paragraph. The pairs can support vision-language grounding of tasks and parts, but they are illustrations, not footage of motion, and rights cover images separately from text.
The buyer's problem
A procedure document that says "align tab A with slot B" next to a labeled drawing contains a small, well-aligned vision-language example. Thousands of such documents could supply grounding data for models that must connect words such as "tab", "latch" or "retaining clip" to what they look like. But a PDF is not a pair list. Figures float away from their text, one drawing serves three steps, captions say "Figure 7" and the step says "see above", and photographs are embedded at low resolution.
Preparation decides whether the pairs are correct. Wrong pairs are worse than none, because they teach the wrong association with confidence. This page describes the method, the schema and the checks. For what illustrated procedures teach in general, see visual work instructions; for scanned inputs, parsing scanned manuals.
The pipeline, step by step
- Page layout analysis. Detect regions: text blocks, pictures, tables, captions, list items, footnotes. DocLayNet, a human-annotated layout benchmark, contains 80,863 manually annotated pages with 11 layout classes across six document categories, manuals among them (Pfitzmann et al., 2022). Models trained on such data, or on the supplier's own, perform this step; accuracy varies by document type, so test on your corpus.
- Figure extraction. Crop each picture at native resolution. Keep vector figures as vector where possible, and record the rendering resolution when rasterized.
- Caption and callout detection. Find the caption, the numbered labels in the drawing and the legend that maps labels to part names.
- Reference resolution. Link "Figure 7", "see below", or implied adjacency to the correct step. This is the step most prone to error.
- Pair construction. Create records tying image, caption, linked step text and callout legend.
- Verification. Human review of a sample, with full review for evaluation material.
Each stage can be automated, and many suppliers use models for stages 1 to 4. The validating LLM-extracted procedure steps checklist covers how to test automated extraction; the same discipline applies to image pairing.
Kinds of pair, and what each teaches
| Pair type | Example (synthetic) | What it grounds | Typical weakness |
|---|---|---|---|
| Figure-caption | "Figure 4. Latch in open position" | Object and state names | Captions are short and generic |
| Figure-step | Step 3 text with its drawing | Action in context | Drawing may show the result, not the motion |
| Callout-label | Number 5 mapped to "retaining clip" | Part-name grounding | Legends are often in a separate table |
| Before-after | Two panels, "before" and "after" | State change | Panels not always labeled |
| Warning pictograms | Hand-crush symbol with caution text | Hazard concepts | Standardized icons, limited variety |
| Photo with annotation | Photo with arrows and text | Real appearance | Fewer than drawings; lower resolution |
The IKEA-Manual dataset shows an unusually rich version of the idea: 102 IKEA objects paired with their assembly manuals, annotated with decomposed assembly parts, assembly plans, manual segmentation and 2D-3D correspondence (Wang et al., 2022). It is research-grade annotation on one document family. Commercial procedure documents will rarely come with 2D-3D correspondence, which is a reason to define up front how deep the buyer's annotation must go.
What this data can support, and what it cannot
Pairs can support:
- Vision-language grounding of part names, tool names and states to appearance.
- Retrieval of the relevant figure for a step, and the reverse.
- Document question answering that depends on diagrams, a gap in text-only corpora.
- Evaluation items such as "which figure shows the correct orientation".
- Annotation seeding for video, where manual figures guide which states to label.
Pairs cannot support:
- Manipulation or control. A drawing shows a state, not the hand motion, force or contact needed to reach it. This matches the general distinction between object recognition data and manipulation training data.
- Real appearance variation. Line art and clean renderings are far from what a robot's camera sees in clutter, lighting and wear.
- Viewpoint matching. Manual figures use isometric or conventional views, not the robot's cameras.
- Proof of correctness in practice. An illustrated step may differ from how workers perform it.
Illustrative pair schema
{
"_comment": "Illustrative - synthetic identifiers and values",
"pair_id": "pr_004127",
"doc_id": "doc_0231",
"revision": "C",
"page": 14,
"image": {
"file": "img_004127.png",
"type": "line_drawing",
"width_px": 1240, "height_px": 880,
"source_resolution": "vector",
"bbox_on_page": [96, 212, 520, 498]
},
"caption": "Figure 4. Latch in open position",
"linked_steps": [{"step_id": "6.3", "confidence": 0.88, "link_basis": "explicit_reference"}],
"callouts": [{"label": "5", "name": "retaining clip", "bbox": [310, 140, 342, 172]}],
"step_text": "Release the retaining clip (5) and lift the cover.",
"language": "en",
"review": {"status": "human_verified", "reviewer": "reviewer_05"},
"rights": {"image_owner": "supplier", "third_party_marks": false}
}
link_basis matters. Values such as explicit_reference, adjacency and model_inferred let a buyer filter to the strongest links for evaluation and accept weaker ones for pretraining.
Acceptance tests
- Pair correctness sample. Reviewers independently judge whether sampled images illustrate the linked step. Set the pass rate before delivery, and set a stricter one for any evaluation subset.
- Link-basis distribution. Report the share of each basis; a corpus resting mostly on adjacency needs a closer look.
- Callout accuracy. Check that label numbers map to the right part names in a sample.
- Resolution floor. Confirm minimum pixel dimensions suit your encoder, and flag upscaled images.
- Duplicate figures. Reused drawings across documents should be flagged to avoid leakage across train and test; split by document family.
- Text-in-image handling. State whether embedded text in figures was transcribed and where.
Limitations and additional preparation
- Domain gap. Procedure illustrations are clean. Pair them with real photographs if the model must recognize parts in the field.
- Scanned inputs. Poor scans degrade layout detection; plan for human review.
- Language. Captions in multiple languages need language tags.
- Redaction. Figures may show proprietary parts, serial numbers, site layouts or people; redaction affects usefulness.
- Annotation depth. Basic pairs are cheap; part-level segmentation or 3D correspondence is a separate, costly scope.
- Visual-to-physical bridge. To connect pairs to action, see SOP-to-video step alignment and the illustrated assembly manuals page.
Permissions and provenance
Figures are often more restricted than text. A manual's drawings may be commissioned from illustrators, supplied by component vendors or licensed from stock sources. Photographs can show workers or customer sites. Ask whether the supplier owns the images or merely holds copies, whether illustrators' contracts allow model training, and whether trademarks or third-party designs appear. The Copyright Office's Part 3 report discusses where copying can implicate rights during AI development; it is background, not a conclusion about a particular license. This is not legal advice.
Questions to ask suppliers
- Which layout model or process finds figures, and how was it tested on this corpus?
- How is each image linked to a step, and is the link basis recorded?
- What share of pairs was human-verified?
- Are callout legends resolved to part names?
- What is the source resolution of images, and are any upscaled?
- Who owns the images, and does the license cover derived pairs?
- Are people, serial numbers or site details visible, and how were they handled?
- Can the pair list be split by document family for evaluation?
Discuss your requirements
If you need image-text pairs drawn from real procedure documents, describe the domains, languages and permitted uses to SourceX. SourceX looks for US businesses that hold suitable documents, checks the data and the supplier's licensing permissions, and coordinates the license and delivery; preparation and annotation are scoped case by case. A request does not guarantee a match, and SourceX follows up about potential sources. See the humanoid training data guide and the procedures hub for the wider map.
Discuss your humanoid training-data requirements with SourceX
Sources
- DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis — layout classes and manual category annotation.
- IKEA-Manual: Seeing Shape Assembly Step by Step — manuals paired with parts, plans and 2D-3D correspondence.
- Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication version) — background on copyright at stages of AI development.