Short answer
Score each procedure on three axes: completeness (are all steps, preconditions and checks present), specificity (are objects, quantities and tolerances named) and executability (could a newcomer or a planner act on each step without guessing). Score sampled documents against written criteria before licensing a corpus, and treat low scores as preparation cost, not only as rejection.
Document-level scoring, not corpus-level diligence
The SOP corpus evaluation checklist asks questions about a whole collection: provenance, deduplication, revisions, rights. This page scores one document at a time. The two work together. A corpus with clean provenance can still be full of procedures that read well and say nothing, and a few excellent documents do not rescue a corpus whose median is poor.
Use the rubric on a random sample from the supplier, not on the documents they send as examples. Two reviewers should score the same documents independently so that disagreement shows where criteria are vague.
Illustrative rubric
Scores are 0 (absent), 1 (partial) or 2 (met). The criteria are examples a buyer would adapt to their task domain; none is a published standard.
| Axis | Criterion | Evidence to look for |
|---|---|---|
| Completeness | Purpose and scope stated | A first line saying what the procedure achieves and when it applies |
| Completeness | Start state and preconditions | Conditions to confirm before step 1 |
| Completeness | All steps present, in order | No jumps in the work a person would have to fill in |
| Completeness | End state and success check | An observable result, not "complete the task" |
| Specificity | Objects named | Items, tools and materials identified by name or ID |
| Specificity | Quantities and parameters | Amounts, settings, times, tolerances with units |
| Specificity | Locations and references | Where things are, what each reference points to |
| Executability | One action per step | Steps do not bundle several distinct actions |
| Executability | Unambiguous wording | Instruction words have one meaning; pronouns resolve |
| Executability | Branches and exceptions | What to do if a precondition fails or a check is negative |
Total possible: 20. Report axis subtotals, not only the sum. A procedure with perfect specificity and no end state is a different problem from one with a good structure and vague nouns.
Completeness
Completeness means that someone who has never done the job can see the whole sequence. Look for the steps that experts skip when writing: setup, gathering tools, confirming a condition, cleaning up and recording results. Documents often describe the middle and leave out the edges.
Check that preconditions and postconditions are explicit; preconditions and postconditions explains why they matter for planning. A procedure that has no way of recognizing that a step succeeded cannot supply success criteria to an evaluation set.
Task decomposition methods help here. Hierarchical task analysis, which Stanton's review traces to training-needs analysis and describes as in continuous use for decades, breaks a goal into sub-goals and plans for performing them (Stanton, 2006). Redrawing a sampled procedure as a small tree is a quick way to find missing sub-goals: any branch that cannot be filled from the text is a gap.
Specificity
Vague nouns and missing numbers are a common weakness to check for in business procedures. "Clean the surface with the appropriate cleaner" depends on knowledge that the writer assumed. For a corpus that will support language grounding, planning or parameter extraction, the document should name what, how much and where.
Check whether the document points to other sources, such as a drawing, a photo, a table or a different procedure, and whether those references are included in the delivery. A step that says "see figure 3" is worthless if figure 3 was stripped.
Executability
A step is executable if the reader can act without asking a question. This is a different property from being correct. Test it by reading the step aloud and asking what a literal-minded newcomer would do. Typical failures: bundled actions ("remove, inspect and replace the filter"), implied agents, undefined terms and steps that rely on judgment ("until it looks right").
Controlled-language standards address this at the sentence level. ASD-STE100 is a controlled natural language and international standard for technical documentation; Issue 9 is dated 15 January 2025 (ASD-STE100). Supplier documents are unlikely to follow it, but its approach of one approved meaning per word and short, direct instructions is a useful yardstick. See controlled-language procedures.
What a high score does not prove
A procedure that scores 20 is well written for a human. It is not evidence that a robot can do the task, and it carries no motion, force or perception information; see what SOPs can and cannot teach a humanoid. Scores also say nothing about whether the procedure matches how work is actually done, since procedures drift from practice and tacit knowledge gaps remain. Finally, the score measures the text; it does not measure rights. A superb procedure with unclear ownership is still a procurement risk.
Running the review
Start with a calibration round: both reviewers score five documents, compare, and rewrite the criteria wherever they disagreed. Then score the sample. Record, for each document, the three subtotals, the reviewer initials and any free-text note on why a point was withheld. Free-text notes become the supplier's improvement list and your preparation estimate.
Report results by document family. A supplier whose maintenance procedures score well and whose housekeeping procedures score poorly has told you which part of the corpus is worth buying. Aggregate scores hide that.
Red flags
- Many documents share identical step text with only titles changed.
- Steps are all one word or all full paragraphs, with no middle range.
- Numbers, quantities or tolerances appear in fewer than a small share of sampled steps.
- Success is defined only as "task complete".
- References to figures, forms or other documents with no linked files.
- Machine-generated text with no human review; see validating LLM-extracted steps.
- The supplier cannot say who last reviewed the document or when.
Using the scores
Set thresholds before scoring, per axis, and decide in advance what happens below them: reject, accept at a lower price tier, or accept with a defined step schema normalization pass. Record the sample size and reviewer agreement in your acceptance file. If the supplier revises documents after sampling, resample.
Questions to ask suppliers
- Can we draw a random sample of documents ourselves rather than receive curated examples?
- Who authored the procedures, and are they reviewed by people who perform the work?
- When was each document last revised, and is the revision history available?
- Do documents include their referenced figures, tables and forms?
- Were any documents generated or rewritten by software, and was the output reviewed?
- Does a style guide or controlled-vocabulary rule apply to the writing?
- How do you handle documents that are known to differ from actual practice?
- Which documents are templates reused across jobs?
- Can you share an anonymized low-quality example so we can calibrate scoring?
Discuss your requirements
If you want a sample scored before you commit to a purchase, tell SourceX what the corpus has to cover, including document types, task domains and permitted uses. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match; SourceX follows up about potential sources. See the humanoid training data guide and the procedures hub, and the site's training data due diligence checklist.
Tell SourceX what your evaluation set needs to cover
Sources
- ASD-STE100 Simplified Technical English — controlled language standard used as a yardstick for unambiguous instructions.
- Hierarchical task analysis: Developments, applications, and extensions — task decomposition method for finding missing sub-goals.