Short answer
Handover and collaborative task demonstrations record two people passing objects or sharing a task, usually from several camera views at once. They show timing, intent signals and who does what, which a robot teammate needs to understand. They are human-human data: they contain no robot actions or interaction forces, and they need consent from both people and capture designed around the pair.
Why two-person interactions are their own data type
A single-worker recording answers "how does one person do this task?" A handover or team task answers a different question: how do two agents coordinate? The giver extends an object, adjusts for the receiver's reach, signals with gaze or a pause, and releases only when the receiver has it. A pair building a frame holds a part while the other fastens it, or one steadies a shelf while the other loads it.
These patterns matter for any robot expected to work alongside people. They also break the assumptions of single-person capture: the camera must see both bodies, labels must say who did what, and consent must cover two people whose faces and actions are linked in each clip.
Task anatomy
- Handover. Reach toward the receiver, orient the object (handle toward receiver), pause or slow, receiver grasps, giver releases, receiver withdraws.
- Hold and operate. One person stabilizes a part, the other drills, fastens, wraps or inspects.
- Joint carry. Two people lift and walk with a shared object, matching pace and turns.
- Tool passing. A surgeon-style call-and-pass is rare outside clinics, but workshops and kitchens have analogues: a cook passes a hot tray, a mechanic passes a wrench.
- Division of labor. Each person owns part of a sequence; timing is flexible, but order matters.
Failure modes include a missed handoff, a release before the receiver has grip, a collision of hands, a part dropped between two people, and mismatched expectations about who moves next. These are the moments that evaluation needs, and they are rare in routine footage; see failure and recovery demonstrations.
What published datasets show
HA-ViD is a human assembly video dataset of industrial-like scenarios with 3,222 multi-view, multi-modality videos, about 1.5 million frames, 96,000 temporal labels and 2 million spatial labels. It includes human-robot shared annotations and analyses of task collaboration, process efficiency and human intention (Zheng et al., 2023). It is an example of annotation designed for collaboration in assembly rather than single-person action labels.
Ego-Exo4D records skilled activities with an egocentric camera and multiple exocentric cameras simultaneously, across 740 participants, with expert commentary (Grauman et al., 2023). Its activities are mostly individual skilled performance such as sports, music and bike repair, but its multi-view design is the template for capturing a pair: wearable views for each person plus fixed views for the exchange. See egocentric vs exocentric workplace video.
Neither study tells you how much pair data a particular robot teammate needs, and neither provides robot control data.
What the data can and cannot support
| Question | Human-human recordings can help | They cannot |
|---|---|---|
| When and where do people hand over objects? | Show positions, timing and approach paths | Provide a robot's reach and release controller |
| What cues precede a release? | Show gaze, pauses and hand cues, if labeled | Prove which cue a person actually used |
| Who does which step in a team task? | Support role and step recognition | Define optimal task allocation for a robot |
| How do people adapt to a partner's pace? | Show variation across pairs | Capture the force exchanged at the object |
| Intent prediction | Provide labeled instances | Validate behavior with a robot in the loop |
Human-human data lacks the asymmetry of a human-robot pair: people expect robots to move differently, to be slower or less predictable, and they behave accordingly. That behavior needs human-robot data, which is a separate collection. Human-robot interaction on shared tasks is covered in human-robot coworking footage. For standards context, ISO/TS 15066:2016 is a Technical Specification on safety requirements for collaborative industrial robot systems, supplementing ISO 10218-1 and ISO 10218-2 (ISO); it addresses robots, not human-human data, but a buyer's safety engineers will want handover scenarios to be consistent with the collaboration modes they design for.
Capture design for a pair
- Views. One head-mounted camera on each person, plus two or more fixed views covering both bodies and the exchange zone; see multi-view synchronized task recording.
- Synchronization. All cameras on a shared clock, with a sync event at the start.
- Identity. Persistent pseudonymous IDs for each person, so labels refer to "giver" and "receiver" by ID and role.
- Pairing. Record the same task with different pairs, and the same person in both roles.
- Objects. Vary mass, size, shape, fragility and handle orientation.
- Pace. Include natural and slowed versions; slowed ones are closer to what a robot can do safely.
- Instrumentation. Optional tools such as force-sensing objects or wrist IMUs; see body-worn IMU demonstrations.
- Settings. Workshops, kitchens, packing benches and assembly stations, as in assembly demonstration video.
Labels
Label per exchange: giver and receiver IDs, object ID, approach start, grasp contact, release, withdrawal, success or failure, and handle orientation. For team tasks add role per step and dependency between steps. Two-person annotation takes longer than single-person annotation, since annotators must track both agents; budget accordingly. See also bimanual task demonstrations for how labels differ when one person uses both hands.
Safety and realism in staged exchanges
Handovers invite hazards: sharp tools, hot items, heavy objects. Agree with the site which objects may be passed and how risky variants are handled; labeled safe and unsafe examples are covered in safety behaviors in demonstrations. Staged exchanges also tend to be more deliberate than real ones. Include some unrehearsed sessions inside real workflows, even if they are harder to annotate, and record which clips are which.
Consent for pairs
Each recording links two identifiable people. Both must agree, and each should know who the other is and what the data will be used for. Consider what happens if one person withdraws: the clip includes both. Consent forms should cover the recording, the pairing, the stated uses and deletion handling. Consent from an employer does not replace consent from the individuals. This is not legal advice; ask counsel. See worker consent and identifiable people and recruiting demonstrators.
Illustrative specification
# Illustrative only - synthetic values
collection: handover_pairs_v0
pairs: 12 pairs, roles swapped in half the sessions
objects: [mug, tray, drill, small_box, tool_A]
views: [head_mounted_giver, head_mounted_receiver, fixed_left, fixed_right]
sync: shared clock plus clap event
exchange_labels: [approach_start, contact, release, withdraw, outcome]
failure_cases: "included when they occur; not staged unless site approves"
metadata: [pair_id, giver_id, receiver_id, object_id, site_id, pace_class]
consent: "both participants, per session; withdrawal applies to whole clip"
Questions to ask suppliers
- How many pairs, individuals and sites are involved, and how were pairs formed?
- Do the same people appear in both giver and receiver roles?
- How many camera views cover each exchange, and are all clocks synchronized?
- Which objects and settings appear, and how varied are handle orientations and masses?
- Are failed or fumbled handovers included, and how are they labeled?
- Who annotated the exchanges, and how was agreement between annotators measured?
- How does consent cover both participants, and what happens when one withdraws?
- Was the pace natural, rehearsed or slowed, and is that recorded per clip?
- Are the participants employees, hired demonstrators or both, and does that change the rights position?
Discuss your requirements
If your robot needs to work with people, describe the tasks, pairings and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources, and can scope a commissioned collection of new recordings if no existing asset fits; a request does not guarantee a match. For wider navigation, see the workplace task demonstration hub and the humanoid training data guide. The catalog entry on egocentric video of manual work is related.
Describe the workplace demonstration data your robotics project needs
Sources
- ISO/TS 15066:2016 Robots and robotic devices — Collaborative robots — safety requirements for collaborative robot systems.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — simultaneous egocentric and exocentric multi-view recording.
- HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding — multi-view assembly video with human-robot shared annotations.