Short answer
Evaluate an SOP corpus on a stratified sample before licensing. Check coverage of your target tasks, step specificity, revision currency, near-duplicates, parse quality of tables and images, evidence that procedures match practice, authorship and confidentiality, and whether the supplier can document the corpus. Set pass thresholds per check in advance, request the evidence behind each, and treat missing revision data or unknown authorship as red flags.
Where this checklist fits
Procedure corpora are easy to oversell. A supplier can truthfully say it holds "4,000 SOPs" when half are superseded, a third are near-copies across sites and many are one-page forms. This checklist is for the decision to license, made on a sample before contract. It assesses the corpus as a whole.
Two neighbors handle other stages. Procedure quality scoring rates individual documents for completeness and executability, and validating LLM-extracted steps checks the output after delivery. What procedure text can and cannot feed in a humanoid stack is settled in the procedures cluster hub; this page assumes you already know you want procedures for planning, grounding or evaluation.
Drawing the sample
- Ask for the inventory first. Every document's ID, type, title, site, department, revision, effective date and file format, without the content.
- Stratify. Sample across sites, document types, departments and age bands. Include the oldest documents and the most frequently used ones.
- Ask for whole documents. Excerpts hide missing preconditions and broken tables.
- Fix the sample size and thresholds before reading. The numbers below are illustrative; set your own to fit the corpus and the intended use.
The checklist
A. Inventory and documentation
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Inventory completeness | Every file has ID, type, revision and effective date | Inventory export from document control |
| Corpus documentation | Purpose, composition, how documents were gathered and recommended uses are written down | A datasheet for the corpus |
| Quality framework | Supplier can describe how document quality is maintained over time | Document-control procedure |
A datasheet should cover motivation, composition, collection process and recommended uses, the structure proposed in Datasheets for Datasets (Gebru et al., 2018); see also datasheet for datasets. ISO/IEC 5259-1 frames data quality as something assessed and improved across the data life cycle (ISO/IEC, 2024), which is the right lens for a corpus that keeps changing.
B. Coverage of target tasks
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Target tasks covered | At least 80% of your target tasks have one or more current procedures | Mapping table, task to document IDs |
| Depth per task family | More than one site or author for priority tasks | Counts by task family and site |
| Exceptions present | Fault, exception and emergency procedures exist for covered equipment | List of non-routine procedures |
Normalizing task names helps. O*NET maps occupational task statements to detailed work activities (O*NET Resource Center), a neutral vocabulary for comparing a supplier's titles with your target list. The full gap-analysis method is on procedure coverage mapping.
C. Step specificity
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Steps name object and target state | 60% or more of sampled steps | Your count on the sample |
| Success checks present | Each procedure ends with a verifiable check | Sample review |
| Parameters stated or referenced | Values given, or a referenced card is included | Sample review |
| Decision points marked | Branches are explicit, not buried in notes | Sample review |
D. Currency and version control
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Current versions identifiable | Superseded documents flagged or excluded | Status field in inventory |
| Revision history | Available for priority documents | Change logs |
| Review cycle | Documents reviewed within the supplier's stated cycle | Last-review dates |
E. Duplication and consistency
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Near-duplicates | Rate measured and reported, not discovered later | Similarity report or your own check |
| Conflicting versions | Same task written differently across sites is flagged | Site-by-site comparison |
Multi-site corpora are full of near-copies, which inflate counts and can leak across train and test splits. Deduplicating procedures covers methods.
F. Format and parse quality
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Native vs scanned | Share of scans known; scans legible | Format counts |
| Tables, callouts, multi-column pages | Steps survive extraction in a trial parse | Trial extraction on the sample |
| Embedded images | Legible and still meaningful after redaction | Sample pages |
Layout is where extraction fails. DocLayNet offers 80,863 manually annotated pages with 11 layout classes across six document categories, manuals among them (Pfitzmann et al., 2022), which shows how much structure a parser must recover. Run a trial extraction rather than trusting file counts.
G. Practice match
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Execution evidence | Completed checklists or sign-offs exist for sampled procedures | Record samples |
| Deviation records | Deviations are logged and linked to procedures | Deviation log extract |
| Observation | A short walk-through or video confirms key steps | Observation notes |
Regulated plants make this easier. Under US drug manufacturing rules, deviations from written procedures must be recorded and justified (FDA / eCFR), so a deviation log shows where the floor departs from the document. Elsewhere, procedural drift explains how to test the gap.
H. Rights and confidentiality
| Check | Pass criterion (illustrative) | Evidence to request |
|---|---|---|
| Authorship per document | Known for every included file | Author or department field |
| Third-party content | Identified and excluded or cleared | Register of OEM, vendor and consultant content |
| Confidential details | Parameters, customers and suppliers handled | Redaction plan |
Ownership questions are covered in who owns a procedure.
Illustrative scorecard
# Illustrative only — synthetic supplier, sample and results
supplier: supplier_S19 (food manufacturing, 4 sites)
inventory: 3,140 documents; 41% superseded; 12% scanned
sample: 120 documents, stratified by site, type and age
results:
coverage_target_tasks: 0.72 # threshold 0.80 -> FAIL
steps_naming_object_and_state: 0.64 # threshold 0.60 -> PASS
success_check_present: 0.55 # threshold 0.70 -> FAIL
near_duplicate_rate: 0.31 # reported -> PASS (dedupe before delivery)
trial_parse_step_recovery: 0.88 # threshold 0.85 -> PASS
execution_records_available: yes # completed checklists for 2 of 4 sites
authorship_known: 0.93 # threshold 1.00 for included docs -> FAIL
decision: proceed with scope limited to sites 01-02; exclude unknown-author files
Red flags
- No inventory, or an inventory without revision status.
- Counts that include forms, templates and superseded files without saying so.
- Procedures that never name objects or end states ("clean area," "check machine").
- A corpus assembled from several acquired companies with no record of origin.
- Equipment procedures that read like the maker's manual but list no source.
- Refusal to provide whole documents for the sample.
Limitations
A sample can miss problems concentrated in one site or department, so stratify carefully and keep the right to reject on delivery. Practice match can only be partly checked from records. And a corpus that passes every check here still carries no motion, force or perception data: it is fit for planners, grounding and evaluation, not for teaching physical skills.
Questions to ask suppliers
- Can you export the document inventory with type, revision, effective date and owner before we see content?
- How many documents are current, superseded or draft?
- How many sites contributed, and do sites keep separate versions of the same procedure?
- What share of documents are scans, and in which languages are they written?
- Do completed checklists, sign-offs or deviation records exist for these procedures?
- Who authored each document, and which contain equipment-maker, vendor or consultant content?
- Which parameters, customer names or supplier names would you need to redact?
- Will you provide whole documents for a stratified sample we choose?
For the general version of these checks, see the robotics dataset evaluation checklist and the catalog's training-data due diligence checklist.
Discuss your requirements
Share your target task list, the intended model use and the checks you need passed. Describe the SOP corpus you are looking for to SourceX: it looks for US businesses that hold suitable procedures, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX reviews the request and follows up about potential sources; a request does not guarantee a match. The humanoid training data guide covers the other data types.
Discuss your humanoid training-data requirements with SourceX
Sources
- Datasheets for Datasets — documenting motivation, composition, collection process and recommended uses.
- ISO/IEC 5259-1:2024 Data quality for analytics and machine learning — Part 1 — data quality assessed across the life cycle.
- O*NET Database: Task Statements and Detailed Work Activities — task vocabulary for coverage mapping.
- DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis — layout complexity in manuals and other documents.
- 21 CFR 211.100 Written procedures; deviations — recorded and justified deviations from written procedures.