Short answer
Build a matrix with target robot tasks as rows and candidate procedure documents as columns, then mark whether each document covers each task fully, partly or not at all, and at which step granularity. The empty and partial cells are your gap list. Run it on a supplier's sample before licensing, and use it to write what the supplier must still find.
Why a coverage matrix
A buyer rarely knows what they are missing until they try to use a corpus. Document counts say little: ten thousand procedures can cluster around five tasks, while a task you care about appears in one paragraph of an unrelated document. A coverage matrix forces the question, for each task you want to support, which documents actually describe it and how well.
The scoping of rows is covered in using occupational task taxonomies. This page assumes you have a task list and need to compare it with real documents. Document-level quality is a separate question, scored with the procedure quality rubric.
Building the rows
Rows are target tasks, written at one consistent level. Two sources help.
- Public task statements. O*NET provides task statements and Detailed Work Activities, with a file mapping tasks to activities (O*NET Resource Center). Mapping your robot's intended duties to activities gives labels others can check.
- Hierarchical task analysis. For tasks you define yourself, decompose each goal into sub-goals and plans. Stanton's review describes the method as a core ergonomics approach in continuous use for over 30 years, originally developed to identify training needs (Stanton, 2006). The leaves of the tree are candidate rows.
Pick the row level deliberately. Rows at job level ("clean a guest room") hide gaps; rows at action level ("pick up a towel") explode. A middle level, such as sub-tasks of a few minutes with a recognizable outcome, tends to be workable, and step granularity discusses the trade-off.
Building the columns
Columns are documents or document groups. Group by business unit, site, equipment type or procedure family so the matrix stays readable. Deduplicate first; near-identical copies inflate the apparent coverage, and deduplicating procedures describes the issue.
Illustrative coverage matrix
Codes: F = fully covered (all steps, preconditions, success check), P = partial (some steps or no check), M = mentioned only, 0 = absent. Values are synthetic.
| Target task | doc_group_A | doc_group_B | doc_group_C | Gap note |
|---|---|---|---|---|
| T01 restock shelf | F | P | 0 | Need success criteria |
| T02 reset table | P | 0 | M | Only one site |
| T03 empty bin | 0 | 0 | 0 | No document |
| T04 open container | M | M | 0 | Mentioned inside other tasks |
| T05 label item | F | F | P | Covered |
For each cell, record a short note when the code is not F. The notes become the supplier brief.
Illustrative specification of the mapping output
# Illustrative only - fields and values are examples
mapping_version: v0.2
task_list_source: "internal HTA tree + O*NET activity mapping (release recorded)"
rows:
- task_id: T03
description: "empty a bin into a container"
level: subtask
required_coverage:
steps: true
preconditions: true
success_check: true
exceptions: ["bin stuck", "container full"]
min_independent_sources: 2
sites_required: 2
cells:
- {task_id: T03, doc_id: docgrp_A, code: 0}
acceptance:
full_coverage_share_target: "set by buyer per priority tier"
unmapped_documents: "listed, not discarded"
reviewers: 2
Notice min_independent_sources and sites_required. A task described by one document at one site is a single point of failure for generalization, and the same applies to evaluation. If you intend to hold out a site, you need enough distinct sources for each task to do it.
Prioritize rows before scoring them. Not every task deserves the same coverage depth, and task frequency for prioritizing procedures describes how to weight rows by how often and how critically a task occurs. Put the priority tier in the matrix so a gap in a high-priority row is not averaged away by good coverage of trivial ones.
Reading the gaps
Empty cells have different meanings and different remedies.
- The task is not documented anywhere. Many routine tasks are learned by watching. A supplier cannot license what does not exist. Options are commissioning a written procedure, or using procedures as collection scripts for new demonstration recording.
- The task is documented, but only the written parts. Many steps depend on knowledge nobody wrote down; see tacit knowledge gaps in SOPs. A fully coded row can still hide the steps that matter most.
- The task is documented but at the wrong level. A whole-job SOP may mention a sub-task in one sentence. You may need annotation to split it out.
- The task is documented but not delivered. The supplier holds more than the sample shows. Ask.
- The task is covered by a different artifact type. A checklist or training slide may carry the content; see SOPs vs work instructions vs checklists.
Also compute the reverse view: documents that map to no target task. They may still be worth having, or they may be filler. Keep them listed so that no one mistakes a large corpus for a well-covered one.
Limitations
The matrix measures text coverage, not capability. A full row means documents describe a task; it does not mean a model trained on them can perform it, and it tells you nothing about motion or force. For tasks where steps matter physically, pair the matrix with a demonstration plan; compare the demonstration coverage plan and the alignment work in SOP-to-video step alignment.
Coding cells is subjective. Two reviewers will often disagree on partial coverage, so keep written rules and report agreement. The matrix is also a snapshot: procedures change, and procedural drift makes coverage decay as practice moves on. Coverage is also not balance. Report, for each row, the number of independent sources and sites, since a cell filled by three near-copies from one site is thinner than it looks. Finally, mapping documents to your tasks can reveal a supplier's business details, so check what you are allowed to retain from a sample.
Questions to ask suppliers
- Can you share a sample drawn across your procedure families so we can map it to our task list?
- How are your documents organized: by role, site, equipment, product or process?
- Which of the following tasks have no documentation in your holdings, even informally?
- For tasks you cover, how many independent documents and how many sites back each one?
- Are there tasks described only inside longer procedures, and can you index them?
- Do you hold checklists, job aids or training slides for the same tasks?
- How often are documents revised, and do you keep past versions?
- Are there tasks where actual practice differs from the documented procedure?
- Would you be able to supply additional documents or annotations if we find gaps?
Discuss your requirements
If you have a task list and need to know whether US businesses hold procedures that cover it, describe the tasks, document types and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match; SourceX follows up about potential sources. Start from the humanoid training data guide or the procedures hub, and the site's page on SOPs, playbooks and knowledge bases.
Discuss your humanoid training-data requirements with SourceX
Sources
- ONET Database (ONET 29.1 per search snapshot): Task Statements and Detailed Work Activities — task statements and activity mapping for matrix rows.
- Hierarchical task analysis: Developments, applications, and extensions — decomposing goals into sub-goals.