Short answer
Safety annotation adds a separate label layer to procedure data: which steps are safety-critical, which hazard each addresses, what control applies, and what must never be skipped or reordered. It needs safety-qualified reviewers and traceable sources. The labels help a planner or evaluator recognize protected steps; they do not make a robot safe or show that a workplace complies with any rule.
The buyer's problem
A planner trained on procedure text sees all steps as equal. In practice some steps exist only because skipping them hurts someone: isolate the energy before opening the cover, verify zero state before reaching in, wait for the machine to stop. If a model reorders, merges or drops such steps while "optimizing", the result may be a plan that looks efficient and is dangerous.
Safety labeling is therefore a different job from step labeling. It uses different reviewers, a different vocabulary and a different failure cost. That is why it is a separate annotation layer, applied on top of the procedure structure in the procedure step schema, not mixed into step text.
Source material for safety labels
| Source type | What it provides | Example |
|---|---|---|
| Job hazard analysis | Task steps, hazards per step, controls | OSHA's booklet describes JHA as a technique that focuses on job tasks to identify hazards before they occur, considering the worker, task, tools and work environment (OSHA 3071) |
| Energy-control procedures | Lockout/tagout steps for servicing | OSHA's standard requires employers to develop, document and use specific procedures to control hazardous energy when employees service equipment; procedures cover scope, purpose, authorization, rules and techniques plus specific shutdown, isolation and securing steps (OSHA, 29 CFR 1910.147) |
| Robot safety standards | Requirements for robots and cells | ISO 10218-1:2025 is edition 3, published February 2025, replacing the 2011 edition, and treats industrial robots as partly completed machinery with inherently safe design, risk reduction and information for use (ISO) |
| Collaborative operation | Human-robot contact limits | ISO/TS 15066:2016 gives safety requirements for collaborative industrial robot systems and their work environment, supplementing ISO 10218-1 and -2 (ISO) |
| Safety data sheets, site rules | Material hazards, PPE | See safety data sheets as handling constraints |
OSHA's lockout/tagout page is partly verified: the regulation text for (c)(4) was not fully retrieved, and a narrow documentation exception exists. Treat the standard as a pointer to a requirement and read the current text before using it as a label source. These sources describe human-worker and industrial-robot safety contexts. None of them says how a mobile humanoid should behave, so label data for a humanoid should not claim that it implements any of them. This is not legal advice.
Labels to specify
# Illustrative only; synthetic content and IDs
step_id: s12
text: "Open the guard and clear the jam"
safety:
critical: true
hazard_types: [stored_energy, pinch_point]
hazard_source_refs: ["jha_031#row4", "loto_proc_007"]
required_before: [s09_isolate_energy, s10_verify_zero_energy]
forbidden_with: [machine_running]
controls: [lockout, guard_in_place_check]
ppe: [cut_gloves, eye_protection]
skippable: false
reorderable: false
criticality: high # scale defined by buyer
evidence_of_completion: [lock_applied_photo, supervisor_signoff]
reviewer_role: ehs_professional
review_status: confirmed
origin: explicit # explicit | inferred
Core fields:
- critical / criticality. A flag plus a level on a scale defined in the spec.
- hazard type and source. Use a fixed list (energy, pinch, chemical, thermal, fall, ergonomic, collision) and a pointer to the document row that supports it.
- required-before and forbidden-with. Hard ordering constraints, the data a planner needs most.
- skippable / reorderable. Explicit booleans, not inferred from wording.
- control type. Elimination, substitution, engineering, administrative or PPE, if your spec uses a hierarchy.
- evidence of completion. What proves a protected step happened.
- origin and review status. Separate documented hazards from annotator inferences.
What the data can support — and what it cannot
It can support:
- Constraint-aware planning. Plans that respect required-before and forbidden-with rules.
- Evaluation of safe behavior. Test items where an agent must refuse, pause or escalate when a precondition for a protected step is missing.
- Language grounding of warnings. Matching "WARNING", "CAUTION" and "NOTICE" text to hazards.
- Prioritizing review. Highlighting which steps need human sign-off. See exception and emergency procedures.
It cannot support:
- A safety case. Labeled data does not establish that a robot or process is safe. Safety assurance for robot systems comes from engineering, risk assessment and standards, not from a dataset.
- Physical safety behavior. How to move near a person, limit force or stop is learned in the robot's control stack. See safety behaviors in demonstrations for what recorded work can show.
- Complete hazard coverage. Documents list known hazards. A hazard no one wrote down is not labeled.
- Site-specific truth. A hazard analysis is valid for a task, equipment and environment at a time. Moving it to another site can mislead; compare against job hazard analyses from each site.
Reviewers and process
Safety labels should be confirmed by someone qualified in the domain: an environmental, health and safety professional, a maintenance supervisor or a safety engineer. General annotators can propose labels, but the final status should come from a reviewer with a role recorded. A workable process:
- Annotators mark candidate steps from warnings, hazard tables and ordering words ("before", "never", "only after").
- A second annotator checks candidates against the hazard source.
- A domain reviewer confirms, rejects or changes each critical step.
- Disagreements are retained with the reasoning, not erased.
- A sample is audited and agreement is reported by hazard type.
Plan for conflicts: two sites may disagree about whether a step is critical. Keep both labels with their sources.
Acceptance criteria
- Every critical step has a hazard type and a source reference or an
inferredflag. - Required-before pairs are consistent with the procedure order, tested automatically.
- Reviewer role and status are present for every critical label.
- Inter-annotator agreement is reported separately for "is this step safety-critical?" and "which hazard?".
- A held-out evaluation set is built from sites or document families not used for training.
- No label claims compliance with a named standard unless the source document does.
Permissions and provenance
Safety documents are often among the most carefully controlled paperwork a business has, and they may include incident history, equipment details and insurer or regulator material. Ask whether the supplier can license them, whether any text comes from third-party standards or vendor manuals, and whether hazard analyses mention named people. Redact numeric limits with symbols so that logic survives; see redacting SOPs and who owns a procedure. Redaction does not settle ownership.
Questions to ask suppliers
- Do you hold job hazard analyses linked to the procedures, or are hazards only in warnings?
- Who reviews and approves safety content, and are those roles documented?
- Are lockout or energy-control procedures separate documents, and what are their revision dates?
- Which hazards appear most often, and are incident-driven revisions recorded?
- Do procedures indicate which steps must not be skipped or reordered?
- Is evidence of step completion (photos, sign-offs) kept?
- Which content is copied from vendor manuals or standards?
- Can sites with different safety rules for the same task be provided separately?
Discuss your requirements
To specify safety-annotated procedure data, describe the documents, hazard types, reviewer qualifications and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. Annotation and safety review are scoped case by case. A request does not guarantee a match; SourceX follows up about potential sources.
See the humanoid training data guide and the procedural data hub. Related catalog entries: SOPs, playbooks and knowledge bases and expert annotations and labels. For hazard-bearing equipment, see equipment operating procedures.
Discuss your humanoid training-data requirements with SourceX
Sources
- 29 CFR 1910.147 The control of hazardous energy (lockout/tagout) — OSHA lockout/tagout requirement for documented energy-control procedures
- ISO 10218-1:2025 Robotics — Safety requirements — Part 1: Industrial robots — ISO 10218-1:2025 safety requirements for industrial robots
- ISO/TS 15066:2016 Robots and robotic devices — Collaborative robots — ISO/TS 15066 safety requirements for collaborative robot systems
- Job Hazard Analysis (OSHA 3071) — OSHA job hazard analysis as a task-focused hazard identification technique