Short answer
Procedural drift is the gap between the written SOP and what workers do. Before relying on a procedure as a robot's task definition, compare it with observation: video, direct walk-throughs, deviation logs and worker interviews. Pass the document only for the steps where practice, record and text agree, and treat the rest as hypotheses.
The buyer's problem
A written procedure is easy to buy and easy to parse. It is also a statement of intent. If a team builds a planner, an evaluation set or a collection script from a document that nobody follows, the robot is aimed at a task the workplace does not perform. The errors are quiet: the plan looks reasonable, the language is correct, and the first on-site trial reveals that the real job skips two steps and adds three.
Drift is normal, not a sign of bad management. Equipment changes, staff find shortcuts, documents lag, and some steps are written for audit rather than for the job. The buyer's task is to measure the gap, not to assume it away.
Why drift happens
Regulated settings write the expectation down. US drug manufacturing rules require that written procedures are followed and documented at the time of performance, and that any deviation is recorded and justified (21 CFR 211.100). The presence of a deviation requirement reflects how normal departures are. A facility with few recorded deviations may be compliant, or may be under-recording.
Ergonomics offers a vocabulary. Hierarchical task analysis is a core ergonomics method, described as over 30 years in continuous use, that was originally developed to identify training needs and has been applied to job aid design and error prediction (Stanton, 2006). Its practice of decomposing a task into goals and sub-goals and then observing real performance against them is the same comparison you need. See hierarchical task analyses for these documents as an asset.
Evaluation checklist
Use this as a diligence list. Pass/fail thresholds are yours to set.
| Test | Evidence to request | Pass looks like |
|---|---|---|
| Revision currency | Effective date, last review date, change log | Document in force during observed period |
| Observation match | Video or walk-through notes for sampled steps | Most sampled steps appear in documented order |
| Record match | Completed checklists and logs for the same jobs | Records consistent with observed order and timing |
| Deviation visibility | Deviation or exception reports and counts | Deviations exist and cite steps |
| Worker confirmation | Short interviews with those who do the job | Workers describe the same steps without prompting |
| Adaptation rules | Written decision points for variants | Common variants are covered |
Each row can yield a per-step label: confirmed, partly confirmed, contradicted or unobserved. Carry these labels into the dataset so users can filter.
Red flags
- The SOP is newer than the observed footage, or older than the equipment seen in it.
- Every sampled job matches exactly: this can indicate scripted or staged recording.
- Records show identical timings across shifts.
- Steps with safety significance are described in the SOP but not visible in any sampled footage or record.
- Workers cannot say where the document is kept.
- A supplier cannot show a single deviation record in a regulated process.
- Documents were rewritten for the data deal rather than taken from the controlled system.
Methods for measuring drift
Direct comparison needs a unit. Use the step. For each sampled job, label which SOP steps occurred, in what order, and which observed actions have no SOP step. Two outputs result: omissions (documented but not done) and additions (done but not documented).
Aligning video to steps is covered in SOP-to-video step alignment. When footage is not available, structured walk-throughs and interviews with workers still reveal additions, and the tacit knowledge gaps page lists the typical places where documents are silent. Records provide a middle level of evidence, and deviation reports show where staff left the text; see completed checklists as execution records.
Illustrative drift annotation
# Illustrative only - synthetic example, not an available dataset
sop_id: sop_0117 (rev r3)
site_id: site_017
sample: 20 jobs, observed by video and walk-through # synthetic
steps:
- step_id: sop_0117_s04
status: contradicted # confirmed | partly | contradicted | unobserved
observed: "Step performed after s06 in most jobs"
reason_given: "Easier to access while door is open"
safety_relevant: false
- step_id: sop_0117_s09
status: confirmed
additions:
- observed: "Operator wipes sensor lens before start"
frequency: frequent
candidate_sop_addition: true
Reason fields matter. A reorder for convenience is different from skipping a guard check, and your data should not treat them the same.
What to do with the result
If the document is mostly confirmed, use it as a task definition and keep the contradictions as annotations. If drift is large, you have three options: observe more and write a revised procedure, collect new demonstrations that define the task directly, or use the document only for terminology and context. New recording involves consent and site access; see existing workplace footage vs commissioned demonstration recording.
Where drift concerns safety-critical steps, do not resolve it by following practice. Escalate to the owner of the procedure; see safety-critical steps in procedures. The broader SOP review list is in the SOP corpus evaluation checklist.
Limitations
A drift check on a sample cannot certify an entire corpus. Observed work reflects one site, season and crew. Workers may behave differently while observed. Video shows what is visible, not why. And it is expensive: every check needs a reviewer who knows the job. Budget for it on the procedures that matter most, not on every document in the archive.
Questions to ask suppliers
- How were procedures written: by operators, supervisors or a quality group, and how recently reviewed?
- Is there recorded footage or photo evidence of the same procedures?
- What deviation or exception records exist, and how are deviations defined?
- Are there known shortcuts, workarounds or unwritten steps that staff would name?
- How often is the SOP audited against practice, and what did the last audit find?
- Which steps are required by regulation or customer contract and cannot drift?
- Can workers or supervisors take part in a short review session on a sample?
Discuss your requirements
If you need documents and observation that agree, describe the tasks, sites and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match; SourceX follows up about potential sources.
See the humanoid training data guide, the procedural data hub, and the catalog entries for SOPs and playbooks and video recordings.
Describe the workplace demonstration data your robotics project needs
Sources
- 21 CFR 211.100 Written procedures; deviations — Drug-manufacturing rules require written procedures to be followed and documented at the time of performance, with deviations recorded and justified.
- Hierarchical task analysis: Developments, applications, and extensions — Hierarchical task analysis is an established ergonomics method originally developed to identify training needs, with applications including job aid design and error prediction.