Using SOPs as Scripts for Commissioned Demonstration Recording

Why script from the supplier's own procedures

Commissioned recording costs site time, demonstrator time and annotation time. If the step vocabulary is invented after capture, annotators argue over boundaries and every site ends up with its own labels. Starting from the business's SOPs fixes the vocabulary before the first take, ties every clip to a named procedure and revision, and makes later alignment to text close to automatic.

A collection script is not the SOP itself, which is written to instruct workers. It is also not the demonstration coverage plan, which sets quotas across sites, people and objects, or the camera specification. The script is the per-procedure session document that sits between them, and the procedures cluster treats it as the bridge from documents to new video.

What published step datasets show about vocabularies

Step structure is easiest when it exists before labeling. COIN organizes its videos in a hierarchy of domains, tasks and steps, with step-level descriptions and temporal boundaries (Tang et al., 2019). CrossTask took an ordered list of steps per task from wikiHow and annotated temporal boundaries only for its primary tasks (Zhukov et al., 2019). Ego4D Goal-Step added hierarchical goal and step annotations to existing Ego4D video, deriving its step taxonomy from the data afterward (Song et al., 2023).

Commissioned recording lets you take the first route deliberately: the SOP supplies the step list, and the script makes each step visible and loggable on camera.

Converting an SOP into a script

1. Pick the revision and confirm it. Use the current approved revision, then walk it with a site supervisor. Note where practice differs; procedural drift explains why this check matters.
2. Assign step IDs. Inherit the SOP numbering, and split or merge only where your model's granularity requires; see choosing step granularity.
3. Classify each step. Record, skip, human-only (for example, signing a log) or restricted (a safety step that must not be staged or shortcut).
4. Write observable cues. For each step, the visible event that starts and ends it, such as "hand contacts lid" or "cart leaves frame," so annotators segment the same way.
5. List and stage objects. Every object and tool with an ID and a staging position, plus which instances rotate between takes.
6. Attach variation axes. Pull the factors for this procedure from the coverage plan: object variants, layouts, demonstrators, hand dominance, lighting.
7. Plan deviation and recovery takes. Omissions, swapped order, dropped items and the correct recovery, each labeled as planned.
8. Set narration rules. Silent, think-aloud, or commentary recorded after the take.
9. Define resets. How the scene returns to the start state, and what is checked before the next take.
10. Run a live step log. An operator taps step IDs on a tablet as they happen, synchronized to the capture clock, giving annotators a first pass at boundaries.

Illustrative collection script

# Illustrative only — synthetic procedure, site and IDs; adapt every value
script_id: SCR_HK_014_v2
source_procedure: {doc_id: SOP_HK_014, revision: 7, site: site_022}
practice_check: walked with supervisor_S3; step 9 done before step 8 on the floor -> both orders allowed
task_goal: "Reset guest bathroom to standard"
steps:
  - id: HK014.06
    text: "Remove used towels and bag them."
    class: record
    start_cue: "first hand contact with a used towel"
    end_cue: "bag tie complete"
    objects: [towel_bath x2, towel_hand x2, mat x1, laundry_bag]
  - id: HK014.07
    text: "Hang fresh towels, bath towels folded in thirds."
    class: record
    start_cue: "fresh towel lifted from cart"
    end_cue: "last towel released on bar"
  - id: HK014.11
    text: "Initial the room checklist."
    class: human_only            # recorded for context, excluded from skill labels
variation_axes:
  towel_color: [white, gray]
  bar_layout: [single_bar, double_bar, hook]
  demonstrator: [operator_A12, operator_B07, operator_C33]
  hand_dominance: [left, right]
deviation_takes:
  - {type: omission, step: HK014.07, recovery: true, share_of_takes: 0.10}
  - {type: order_swap, steps: [HK014.06, HK014.07], recovery: false, share_of_takes: 0.05}
narration: none during take; 60-second commentary afterwards
reset: restore start state from photo REF_HK014_start; checker confirms before next take
step_log: tablet tap per step ID, synced to capture clock
takes_per_combination: set by buyer after pilot
restricted: no chemical decanting on camera without site EHS approval

Field reference

Capture parameters such as camera placement and frame rate belong to the head-mounted camera capture specification, and the labeling rules downstream to SOP-to-video step alignment.

Scripted structure, natural execution

Over-scripting is the main failure. Demonstrators who are told how to move, or who read steps aloud, perform slowly and uniformly, and the footage stops reflecting real work. Give demonstrators the goal and the step list, not motions, and keep some free takes where they work as they normally would.

Published datasets show both ends. Assembly101 recorded unscripted assembly and disassembly, so its mistakes are natural (Sener et al., 2022). CaptainCook4D recorded recipe executions that were either normal or intentionally erroneous, under IRB approval with written consent (Peddi et al., 2023). Planned errors are easy to label but may not look like real ones, so keep both kinds if error detection matters. Post-take commentary can be a lower-friction way to collect language than narration during work; Ego-Exo4D paired simultaneous first- and third-person recordings of skilled activities with expert commentary (Grauman et al., 2023).

Whether to script in a staged workcell or during live operations is a separate decision covered in staged vs live operation recording.

Acceptance tests

1. Script compliance. The share of takes with every mandatory step logged meets your threshold.
2. Log-to-video agreement. On a sample, logged step boundaries fall within a tolerance you set of annotator boundaries.
3. Quota fill. Every variation combination in scope has the planned number of takes.
4. Deviation labeling. Every planned deviation take is flagged, and unplanned errors are flagged separately.
5. Restricted steps. No restricted step was staged or shortcut on camera.
6. Consent. Every demonstrator's consent record is linked to their pseudonymous ID before delivery.

Run these first on a pilot; pilot collection scoping covers go/no-go criteria.

Limitations

A script organizes human video; it does not create robot data. The recordings still lack joint states, commanded actions and forces, and need retargeting, inferred actions or co-training with robot data to inform control; see what human demonstration video can and cannot do. Scripts also inherit the SOP's blind spots, and scripted errors can be unrealistic. Recording on a working site disrupts operations and needs consent from everyone on camera. Sharing the SOP with a collection provider also needs the right to do so; see who owns a procedure.

Questions to ask collection providers

1. Who will walk the procedure with a site supervisor before recording, and how are differences recorded?
2. Can you run a live step log synchronized to the capture clock?
3. How do you brief demonstrators so they follow the step list without being coached on motions?
4. How are planned deviations staged safely, and who approves them?
5. Which steps will site EHS restrict from recording?
6. How do you reset scenes between takes, and how is the reset checked?
7. How will you report quota fill per variation combination during collection?
8. How are consent and withdrawal handled for demonstrators and bystanders?

Discuss your requirements

If you want commissioned demonstrations organized around a business's own procedures, describe the tasks, procedures and variations to SourceX. The site describes a separate physical-world path for new recordings of hands-on work coordinated with businesses, where each opportunity has its own requirements. SourceX reviews the request and follows up about potential sources; a request does not guarantee a match. For context, see the catalog entry for first-person video of skilled manual work and the humanoid training data guide.

Describe the workplace demonstration data your robotics project needs

Sources

• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — hierarchical step structure with temporal boundaries.
• Cross-task weakly supervised learning from instructional videos — ordered step lists defined before annotation.
• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — step taxonomy built after unscripted recording.
• CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — normal and intentionally erroneous executions with written consent.
• Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities — unscripted recordings with natural mistakes.
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — simultaneous ego and exo recording with expert commentary.