Safety Behavior Examples in Workplace Demonstrations

Why a labeled subset

Safety behaviors appear in ordinary work footage, but they are scattered and unlabeled. A worker bends their knees to lift a tote, steps back as a forklift passes, puts on gloves before handling a part. A buyer who wants to test whether a model recognizes, anticipates or respects such behaviors needs those moments pulled out, named and checked.

Two uses are common. One is understanding: can a perception or planning model recognize hazardous or careful behavior in a scene? The other is evaluation: does a policy behave appropriately in scenarios drawn from real practice? Neither makes a robot safe. The robot's safety case rests on its design, controls and standards, not on imitation of human habits.

What the subset can and cannot support

Human video contains no robot joint states, forces or commanded actions, so the subset is observation data. See human demonstration video for robot learning for what that implies in general.

Reference frameworks for labels

Labels can borrow vocabulary from published occupational guidance, provided the buyer treats it as a source of terms and not as a pass/fail test applied to video.

The Revised NIOSH Lifting Equation is a method for estimating the risk of low back disorders associated with manual lifting tasks, including asymmetrical lifts and lifts with less than optimal hand-object coupling (NIOSH, 2021). Its inputs, such as horizontal distance, vertical height, asymmetry and coupling, suggest label fields for lifting clips. Whether a recorded lift gets a risk estimate is a separate ergonomics task requiring measured loads and distances that footage rarely contains.

OSHA's ergonomics overview lists musculoskeletal disorders and names risk factors including forceful exertion, repetitive motion and awkward and static postures (OSHA). Those risk factors are a usable starting taxonomy for posture labels.

For collaborative operation, ISO/TS 15066:2016 is a Technical Specification on safety requirements for collaborative industrial robot systems and their work environment, supplementing ISO 10218-1 and ISO 10218-2 (ISO). It governs robot systems. Using it to label human behavior would be a category error; use it instead to frame what a human-robot scenario test should be checked against by your safety engineers. See also human-robot coworking footage.

Illustrative collection and labeling specification

# Illustrative only - synthetic values, set by the buyer and site safety staff
safety_subset:
  behavior_classes:
    lifting_posture: [neutral, twisted, reaching_far, one_handed]
    proximity: [person_within_zone, person_approaching, vehicle_passing]
    ppe: [gloves_on, gloves_off, eye_protection_on, eye_protection_off]
    handling: [two_person_lift, use_of_cart_instead_of_carry]
    stop_and_check: [pause_before_action, verbal_call_out]
  per_instance_fields: [clip_id, class, start_s, end_s, outcome, annotator_id, confidence]
  context_fields: [site_id, zone_type, load_class (light/moderate/heavy as defined in doc), lighting]
  capture:
    views: [head_mounted, one_fixed_oblique]
    people: "all individuals in frame covered by consent or excluded"
  staging: "unsafe examples staged only if site-approved and risk-free; otherwise omitted"
  excluded: [injuries, near-miss reconstruction without site approval]
  qa: "two annotators on 20 percent; disagreement reviewed by a safety-trained reviewer"

Collection approaches

From existing footage. Mine work recordings for instances of the target behaviors. This yields naturalistic examples, but unsafe examples are rare, and labels must be added. Safe behavior is easy to find; unsafe behavior is mostly absent or edited out. See rare events in demonstrations.

Commissioned and staged. Ask demonstrators to perform both careful and careless variants under site-approved supervision. Staged unsafe behavior must never create real risk; many sites will not allow it, and some negative examples are better obtained from simulation or from synthetic scene editing than from people.

Mixed. Use footage for positives and staging for contrastive pairs. Avoid training the model on deliberately exaggerated staging as if it were natural.

Which behaviors to start with

Pick a small number of behaviors that are frequent, visible and relevant to your deployment. Lifting and carrying are natural starting points because they are common and well described in occupational guidance; see manual load handling and carrying. Proximity and PPE labels need fixed or wide views as well as head-mounted ones.

Unsafe or awkward moments are scarce, so plan how to find them. One route is mining existing footage for slips, recoveries and corrections; see failure and recovery demonstrations. Another is accepting that the evaluation set will be small and curating it by hand. A small, well-labeled, held-out set is usually more informative than a large noisy one.

Quality and acceptance

• Each class has instances from several sites, people and loads, not one demonstrator.
• Annotators are trained on class definitions with examples; agreement is measured.
• A reviewer with relevant safety background settles disputed labels.
• No label implies a legal or regulatory judgment about compliance.
• Evaluation sets are held out by site and person, and not mixed into training; see task-understanding evaluation sets.

Limits to state in the contract

State plainly that the data is not an injury record, not a compliance audit and not a validation of robot safety. Do not let labels like "unsafe" be read as evidence about a particular employer's practices; suppliers may reasonably worry about that, and it affects rights and willingness to share. Footage of near misses or injuries raises employment and liability questions. This is not legal advice; ask counsel. Also confirm workers have agreed to recording; see worker consent and identifiable people and machine safety constraints for recording.

Questions to ask suppliers

1. Which safety behaviors appear in your footage or can be staged, and how often?
2. Who defined the class labels, and what is their safety or ergonomics background?
3. Are unsafe or near-miss examples real, staged or absent, and how were they approved by the site?
4. How many sites, people and load types appear in each class?
5. How were annotators trained, and what is inter-annotator agreement for the hardest classes?
6. Can labels be released without identifying the employer's safety record?
7. Are injury or incident recordings excluded, and how is that checked?
8. What consent covers workers who appear only in the background?
9. Can I hold out entire sites or people for evaluation?

Discuss your requirements

If your evaluation plan needs labeled safety-related behaviors, describe the behaviors, sites and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources, and can scope a commissioned collection of new recordings if no existing asset fits; a request does not guarantee a match. For wider navigation, see the workplace task demonstration hub and the humanoid training data guide. The catalog entry on evaluation sets is related.

Tell SourceX what your evaluation set needs to cover

Sources

• Applications Manual for the Revised NIOSH Lifting Equation (DHHS (NIOSH) Publication No. 94-110) — risk estimation for manual lifting and its input factors.
• Ergonomics - Overview — musculoskeletal disorders and ergonomic risk factors.
• ISO/TS 15066:2016 Robots and robotic devices — Collaborative robots — safety requirements for collaborative robot systems.