First-person video of skilled manual work for robotics and embodied AI

An egocentric video dataset of manual work is first-person footage from a head-mounted camera worn by a skilled worker doing real tasks, showing hands, tools and materials as the worker sees them. SourceX coordinates new recordings with consenting workers at partner businesses in manual-work industries, with annotations and sensors such as IMUs where agreed. This is new collection, not an existing archive, and each program has its own requirements, such as minimum team size and region.

Request a collection program Email the data team

Dataset manifest

New collection program

What it is: First-person video of skilled workers doing real tasks at partner businesses
Typical systems: Head-mounted cameras; wrist cameras, IMUs or other sensors only where agreed
Typical history: New recordings; collected for your order
Modality: Video, optional audio and synced sensor streams, time-stamped text annotations
Delivery formats: Agreed per program; video files with JSON annotations and synced sensor logs
Preparation: Consent recorded; bystander faces, screens and confidential material blurred or cut
Licensing: Permitted use, ownership and any exclusivity agreed before recording starts
Availability: New collection program; per-program requirements such as minimum team size and region; not guaranteed

Request a collection program

What a delivery contains

Fields vary by source system and are fixed per order. A typical delivery includes:

Field	Type	What it holds
clip_id	string	Pseudonymous clip ID, linked to its recording session, task, annotations and sensor files.
worker	object	Pseudonymous worker ID, consent reference and attributes the worker agreed to share, such as trade, experience and handedness.
site	object	Site type, region, workstation and conditions such as lighting and noise, without the business's name or address.
task	object	The task performed, its variant and the reference procedure it follows, where the partner allows that to be shared.
capture	object	Mount position, resolution, frame rate, field of view, stabilization setting and camera calibration ID.
sensors	array	IMU or other sensor streams where agreed, with sampling rate, placement and sync method.
segments	array	Time-stamped action segments with step number and verb-noun labels from the program's vocabulary.
hand_object	array	Contact and release times, which hand, the object or tool, grasp type and, where agreed, boxes or masks.
narration	array	Time-stamped narration, either the worker thinking aloud during the task or commentary recorded afterward.
outcome	object	Whether the task was completed, with errors, corrections and any safety stop noted.
ppe	array	Protective equipment worn, such as gloves or safety glasses, which affects hand appearance and pose estimation.
privacy	object	What was blurred, muted or cut — bystander faces, screens, documents, badges — before delivery.
quality_flags	array	Time ranges with hands out of frame, motion blur, occlusion or sensor dropouts, found in quality review.

Example record

{
  "clip_id": "clip_5e0a91",
  "session_id": "ses_2c7f",
  "program": "prg_panel_wiring",
  "worker": { "id": "wkr_83b1", "trade": "panel_wirer", "experience_band": "10y_plus",
              "handedness": "right", "consent_ref": "cns_83b1_v2" },
  "site": { "type": "control_panel_shop", "region": "[REGION]", "station": "bench_04" },
  "task": { "id": "tsk_terminal_block_wiring", "procedure_ref": "[WORK_INSTRUCTION_ID]" },
  "capture": { "mount": "head", "resolution": "1920x1440", "fps": 60, "fov_deg": 120,
               "stabilization": "off", "calibration": "cal_cam07", "audio": true },
  "sensors": [ { "type": "imu", "placement": "head", "rate_hz": 200, "sync": "shared_clock" } ],
  "duration_s": 912.4,
  "ppe": ["safety_glasses"],
  "segments": [
    { "start": 0.0, "end": 38.5, "step": 1, "verb": "inspect", "noun": "wiring_diagram" },
    { "start": 38.5, "end": 61.2, "step": 2, "verb": "strip", "noun": "wire_end" },
    { "start": 61.2, "end": 84.9, "step": 3, "verb": "crimp", "noun": "ferrule" },
    { "start": 84.9, "end": 102.3, "step": 4, "verb": "insert", "noun": "terminal_block" },
    { "start": 102.3, "end": 106.0, "step": 5, "verb": "pull_test", "noun": "wire", "result": "fail" },
    { "start": 106.0, "end": 139.7, "step": 3, "verb": "recrimp", "noun": "ferrule", "correction": true }
  ],
  "hand_object": [
    { "t_contact": 38.9, "t_release": 84.0, "hand": "left", "object": "wire", "grasp": "precision_pinch" },
    { "t_contact": 39.1, "t_release": 60.8, "hand": "right", "object": "wire_stripper", "grasp": "power" }
  ],
  "narration": [
    { "t": 103.5, "source": "think_aloud", "text": "That one pulled out. Ferrule's short, redo it." },
    { "t": 140.2, "source": "think_aloud", "text": "Tug test on every one before I dress the duct." }
  ],
  "outcome": { "completed": true, "corrections": 1, "safety_stop": false },
  "privacy": { "faces_blurred": 1, "documents_blurred": 1, "audio_muted": [[512.0, 524.5]] },
  "quality_flags": [ { "start": 430.0, "end": 433.5, "flag": "hands_out_of_frame" } ]
}

Synthetic record for illustration. Field names, structure and format are agreed per order.

What AI teams use it for

Learn manipulation from human demonstrations

Close first-person views of hands, tools and parts during real tasks support visual pretraining, affordance learning and retargeting human motion to robot hands.

Train video and world models on procedures

Long recordings of multi-step tasks show how work unfolds over minutes, including tool changes, checks and corrections, rather than short isolated actions.

Recognize steps and catch mistakes

Step-level segments train models that track where a worker is in a procedure, flag skipped or out-of-order steps and anticipate the next action.

Model hand-object interaction and grasps

Contact times, grasp types and object labels supervise hand pose and grasp models in real clutter, with gloves, tools and deformable materials.

Evaluate embodied and video-language models

Held-out recordings from trades and worksites that public datasets rarely cover test whether a model's grasp of manual work carries over to new settings.

Use-case guides: Robotics and embodied AI, Private evaluation sets

What makes this data valuable

Skilled performers

Experienced workers doing their own jobs show efficient, safe technique that novices and staged demos miss.

Errors and recoveries labeled

Re-dos and corrections are marked rather than cut, so a model sees how an expert notices and fixes a slip.

Calibrated, synced capture

Known camera intrinsics and sensor sync make 3D hand and head motion recoverable.

Fixed annotation vocabulary

A written guide, a set verb-noun list and agreement checks keep labels consistent across annotators.

Variety across people and sites

Several workers and sites performing the same task keep a model from learning one person's habits.

Real conditions

Actual lighting, clutter, noise and protective equipment, instead of a tidy lab bench.

What commissioning footage changes

Commissioned footage is shaped around your model from the first session, which no existing dataset can be. Public egocentric datasets were mostly recorded for academic research, lean toward daily-life activities, and come with their own task lists, label vocabularies and license terms, some of which restrict commercial use. Building on them means accepting choices someone else made for a different purpose.

In a collection program those choices are made for your use. The task list follows the work your robot or model will face, the annotation vocabulary matches your action space, consent and permitted use cover commercial training before anything is recorded, and because the footage has never been published, part of it can be held out as a clean evaluation set. The trade-off is that the data does not exist yet: it has to be planned, piloted and recorded, and it depends on partner businesses that meet the program's requirements agreeing to take part.

Choices that shape what the footage teaches

The number of different performers matters as much as the amount of footage. Recordings from one or two workers teach their personal habits rather than the task, which is one reason a program's minimum team size matters: the partner needs enough people doing the work for that variety to exist.

When to record is a trade-off. Filming during real jobs gives authentic pace and variation but little control over which tasks appear, while dedicated sessions control the task mix but drift toward demonstrations. Workers who know they are being filmed may also work more carefully or more by the book than usual, an effect that can fade as cameras become routine over a longer program.

Narration has a similar tension. Talking through a task while doing it captures in-the-moment reasoning but can slow some workers down or change how they work. Commentary recorded afterward, while the worker watches the clip, avoids that, but tends to explain the work rather than report it as it happened. Some programs use both.

Audio is a privacy decision as much as a technical one. It adds narration and tool sounds that help with step recognition, but it also records co-workers and customers, so it is often turned off, or kept with other people's speech muted.

What to check before licensing

Read the consent materials. They should name AI training and the agreed license and explain withdrawal. Ask how participation was kept voluntary, for example separate from performance reviews.
Ask how people who did not consent are protected — filming zones, notices, scheduling, face blurring and cutting segments — including customers at their own premises.
Check sample clips for identifying details beyond faces, such as voices, tattoos, name badges and reflections in glass or polished metal.
Confirm the site owner approved recording where it differs from the employer, and how screens, documents and proprietary processes are kept out of frame.
Review the safety assessment. Mounts must not interfere with protective equipment or create snag hazards, and workers must be able to stop recording at any time.
Measure on the pilot how often hands are in frame and how much motion blur there is, and check video-to-sensor sync drift.
Confirm stabilization is off or documented. Electronic stabilization crops and warps each frame, which undermines the fixed camera calibration that 3D hand tracking relies on.
Agree retention, deletion on withdrawal and whether the partner business may also use the footage, before recording starts.

How licensing works through SourceX

1
Define
Send the domain, modality, volume, format, timeline and permitted use you need.
2
Source
SourceX identifies businesses that hold matching data and are open to licensing it.
3
Qualify
Fit, rights and quality are checked, and you review samples before committing.
4
License
Scope, permitted use, exclusivity, price and obligations are agreed in writing.
5
Deliver
Approved data is prepared, de-identified where required and transferred securely.

Questions buyers ask

Is this an existing archive of first-person video?

No. This is new data collection, not licensing an existing archive: recordings are made for your order with partner businesses in manual-work industries, under a capture plan, consent process and annotation guide agreed in advance. General CCTV footage and arbitrary photos do not qualify: they do not show the work from the worker's viewpoint, and they are rarely captured with consent that covers this use.

Which industries and tasks can be recorded?

That depends on which partner businesses take part. Candidate settings include assembly lines, repair and maintenance shops, the building trades, warehouses and commercial kitchens. Each program has its own requirements, such as a minimum team size and region. Describe the tasks, environments and regions you need; a program runs only if suitable partners agree to take part, so it is not guaranteed.

How is consent handled for workers and bystanders?

Workers who wear cameras give informed consent that covers AI training and the agreed license, and the license sets out what happens to delivered data if consent is withdrawn. Recording is planned to keep other people out of frame, and faces captured incidentally are blurred. Consent and workplace-monitoring rules differ by country, so the consent process is set up for each program's region.

What annotation options are there, and how is label quality checked?

Annotation is agreed per program. Common options are action segments with step labels, hand-object interactions with contact times and grasp types, object boxes or masks, narration from the worker, and task outcomes with errors and corrections. Labels follow a written guide: a pilot batch is labeled and reviewed before full annotation starts, and a share of clips can be labeled twice to measure agreement between annotators.

Can IMUs or other sensors be recorded with the video?

Yes, where the partner and workers agree and the site's safety rules allow. Head or wrist IMUs, a second camera, eye tracking or depth sensors can be added. Each extra sensor needs documented placement, calibration and a sync method, and has to stay out of the way of the work and of protective equipment.

How are confidential worksites and safety handled?

Each site is reviewed with the partner before recording. Areas, screens, documents and proprietary processes that cannot be filmed are excluded or blurred, and recording at a customer's premises needs that customer's permission. Cameras and mounts must not interfere with protective equipment, workers can pause recording at any time, and hazardous tasks are recorded only with the site's safety sign-off, or not at all.

What determines the cost and timing of a collection program?

There is no public price list or standard timeline. Both depend on the tasks, the number of workers and sites, the amount of footage, sensors and annotation depth, the regions involved, and how quickly partners that meet the program's requirements can be found and their workers brought through consent. The request form asks for a budget range and timeline so the program can be sized to them.

Field service and maintenance work orders
Work orders tracing symptom, diagnosis, parts and fix, with asset histories and photos
Manufacturing quality and inspection records
Inspection results, nonconformances, dispositions and corrective actions from production plants
SOPs, playbooks and internal knowledge bases
Written procedures with page history, ownership and links to execution records
Construction project records
RFIs, submittals, change orders, daily logs, inspections, photos and schedules from building projects

Evaluating this data for procurement?

Diligence packets are prepared per dataset. Rights, privacy processing and quality differ between datasets.

Request dataset diligence

Tell us what your models need

Send your spec — domain, volume, format, timeline and permitted use — and SourceX will match it against partner data and come back with what can be licensed.