Skip to content

Systems and records

Physical AI human correction data: capturing a worker correcting a robot

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Capture a worker correcting a robot as a bounded event inside a continuous recording: a pre-trigger window showing the robot's error, logged pause, takeover and return times on one clock, a per-frame flag for who was in control, recovery and correction marked separately, synchronized robot state, actions and video, a reason code and the task outcome. Settle worker notice and licensing rights first.

Video and sensors

Physical AI human correction

This page covers physical AI human correction data: capturing a worker correcting a robot.Illustrative

Key takeaways

  • The value of a correction is the record of what the robot got wrong and what the worker did instead, not the video of a person.
  • Log pause, takeover and return times and a per-frame control flag at the controller, not by reviewing video later.
  • Mark recovery and correction separately so a model does not learn detours as part of the task.
  • E-stops and safety stops belong in safety review, not in imitation data.
  • Worker notice, ownership of robot logs and the right to license are three separate questions to settle first.

How should a physical AI dataset capture a worker correcting a robot?

Capture the correction as a bounded event inside a continuous recording: a few seconds of context before the worker stepped in, the exact moments control passed to the worker and back, every frame tagged with who was in control, the worker's recovery and correction separated, a reason code, and the task outcome afterward. Synchronized robot state, actions and video make the event usable; a clip of a person fixing something does not.

This matters because the correction is the one moment where the data shows what the robot got wrong and what a skilled person did instead. Robotics teams train on these moments to teach a policy to recover. A business hosting robots on its floor, whether a warehouse, a machine shop or a lab, is often already generating them without recording them in a form anyone can use.

The rest of this page sets out the permissions to settle first, a field-level capture specification, the synchronization and coverage checks that decide whether the data holds up, an illustrative event, and the cases where a correction should not become training data at all.

Why robotics teams value a correction more than another clean demonstration

A policy trained only on successful demonstrations has never seen the states it drifts into when it makes a small mistake. Hugging Face's LeRobot documentation describes this as distribution shift: small errors compound and push the robot into states that were not in training, and human-in-the-loop collection addresses it by recording the human's recovery and correction when the robot is about to fail.

The approach has a research lineage. HG-DAgger, an interactive imitation learning method by Kelly and co-authors, lets a human expert watch the learner and take control only when needed, then hand control back. The LeRobot documentation describes the same pause, takeover and return-to-policy loop, with the policy and the human able to alternate control several times in one episode.

For a business on whose floor this happens, the implication is simple. The value is not the video of a worker. It is the precise record of where the robot was, what the worker changed and whether the task then succeeded.

Settling permissions, notice and task boundaries before recording starts

Decide four separate things before any correction is captured: who may be recorded and how they are told, who owns the robot logs and video, what the business may license, and which tasks and areas are in scope. These are different questions. A worker notice does not give the business licensing rights, and a robot vendor's contract can restrict what the business may do with logs the robot produced.

Workplace monitoring notice rules vary by state and are worth reading in their own words. Connecticut's statute requires an employer engaged in electronic monitoring to give prior written notice of the types of monitoring to employees who may be affected, with a conspicuous posting counting as that notice. Public Act 26-73, effective October 1, 2026, adds that the notice must name the specific workplace locations where monitoring may occur, with exceptions such as monitoring for security and employee safety. New York's Civil Rights Law section 52-c requires notice on hiring, with employee acknowledgment, for monitoring of telephone, email and internet use, which shows how differently scoped these laws are. Counsel should map the rules for each site.

Then draw task boundaries. Name the robot cell, the tasks in scope and the camera fields of view, and keep areas outside the task, such as break rooms and other workstations, out of frame. Frame cameras on the work surface and hands where the task allows; a face rarely adds anything to a correction event and adds privacy obligations.

The correction event capture specification

Use these fields for every correction event. Field names are suggestions; what matters is that each one exists, has a defined source and is filled the same way across shifts and sites. Version the specification and record its version on every event.

  • Identify the worker by a role and a pseudonymous operator code, never by name, and keep the code key inside the business.
  • Record the clock source and measured offset for each stream on every episode.
  • Store the conditions that vary: shift, lighting, part variant, fixture, and robot software version.
The correction event capture specification
FieldDefinitionSource signalCommon failure
Event and episode IDsA unique event ID nested in the episode of the task attempt it belongs toRecording software; episode counterEvents split into separate files with no shared episode ID
Task and boundaryTask name, cell, start and end conditions, spec versionWork order or task definitionFree-text task names that differ by operator
Pre-trigger windowRecorded frames before the takeover, enough to show the robot's error developingRing buffer in the recorderRecording starts only when the worker acts, so the cause is missing
Trigger typePause input, physical takeover, verbal stop, safety stop or e-stopTeleop input, controller events, safety PLCAll stops logged as one generic interruption
Control holder per frameWhether the robot policy or the human was in control at each frameIntervention flag written by the recorderFlag inferred later from video, with no logged signal
Handoff timestampsPause, takeover and return-to-policy times on one clockRecorder eventsHandoffs taken from a separate camera clock without an offset
Recovery and correction segmentsRecovery returns the robot to a good state; correction shows the right behavior from thereOperator marker or post-hoc annotationBoth merged, so the policy learns detours as the task
Robot state and action streamsJoint states, gripper, commanded actions at a stated rateRobot controller topics or dataset framesActions logged at a different rate with no resampling note
Camera metadataCamera ID, mounting position, resolution, frame rate, calibration dateCamera configurationA camera moved between shifts and nobody recorded it
Reason, outcome and reviewWhy the worker stepped in, whether the task then succeeded, and safety review statusOperator reason code, quality or MES record, reviewer sign-offOutcome left blank or set before rework finished

Do the clocks agree, and does each episode end where it says it ends?

A correction is only useful if the moment the worker took over lines up with what the robot was doing at that instant, so clock alignment is a capture requirement, not cleanup. Check it on every episode before anything else.

ROS 2's rosbag2 describes a bag as a file full of timestamped messages and distinguishes the time a message was received by the recorder from the time it was published; playback orders messages by reception time by default. With simulated time enabled, recorded messages take the latest value on the clock topic, and nothing is written until the first clock message arrives. Record which time base each stream used, because mixing them can shift a handoff by enough to misattribute the error.

Episode boundaries need the same care. The RLDS format used by many robot learning datasets marks the first and last step of every episode and distinguishes a terminal step from a truncated episode that simply stopped. A correction episode that ends because the shift ended is truncated, not successful, and the outcome field should say so.

LeRobot's human-in-the-loop recording writes an intervention flag on each frame, tagged true while the human is correcting. If your recorder does not log a flag like this, add one at the controller level rather than inferring control from video afterward.

What variation and recovery behavior is missing from the footage?

Correction data is selected by the people watching, so it over-represents failures that a worker noticed in time and under-represents everything else. Before offering it, compare what the set contains against what the floor actually does.

Look for these gaps. Pauses may not be recorded at all: the LeRobot protocol notes that no frames are recorded during a pause, so the seconds when the worker decided to step in exist only in the pre-trigger window. Failures that happened while nobody watched, or that an e-stop ended, may carry no correction at all. Night shifts, uncommon part variants, worn fixtures and new operators are often thin.

Report coverage as counts by condition, not as a single total. A set that is, to take an illustrative figure, 90 percent day-shift corrections on one part family should say that in its documentation, so the buyer can decide whether to request more.

Worked example (illustrative): a worker reorients a part in a bin-picking cell

The event below is invented to show the specification filled in. A pick-and-place robot in a packaging cell repeatedly grasps a molded part by a thin edge; the part slips. A trained operator pauses the robot, guides the arm back to a clear pose, regrasps the part by its body and returns control.

  • Result, per the checks in the table below: the event is kept, with the side camera cropped to the work surface.
  • Exception: a second event in the same episode ended in an e-stop when a person entered the cell. It is routed to safety review and excluded from the correction set, even though it also shows a human stopping the robot.
  • Limit: one operator made all corrections on this variant, so the set shows one person's technique. The documentation says so.
Worked example (illustrative): a worker reorients a part in a bin-picking cell
FieldIllustrative valueCheckResult
Event and episodeEvent 0147 inside episode 2210 of task 'pack variant B'IDs present in all streamsPass
Pre-trigger windowSix seconds buffered before takeover, showing two slipping graspsRobot error visible before the triggerPass
Trigger and handoffsFoot pedal pause at 41.2 s, takeover at 42.0 s, return at 55.6 sAll on the recorder clockPass
Control holderIntervention flag true from 42.0 s to 55.6 sFlag matches handoff timesPass
SegmentsRecovery 42.0 to 47.3 s; correction 47.3 to 55.6 s, marked by operatorSegments do not overlapPass
CameraOverhead camera framed on the tray; one side camera shows operator torsoNo faces in framePass after cropping the side view
OutcomePart packed; quality record shows no rework on the lotOutcome from the quality systemPass

Corrections that should not become imitation data

Not every human intervention shows correct behavior. A model trained on a correction learns to copy it, so the set should only contain corrections a qualified reviewer would endorse.

Exclude or separately label these: safety stops and e-stops, which are incident evidence for safety review rather than technique to imitate; corrections by untrained or visiting staff; corrections later reversed by a supervisor or followed by rework; and takeovers where the worker was improvising around a fixture fault rather than correcting the robot. Keep them in a separate failure log if they have value for evaluation, with the reason they are excluded from training.

Signs it is time to stop collecting and involve a privacy or safety specialist

  • Workers or their representatives raise concerns about being recorded, or the notice posted at the cell no longer matches what the cameras capture.
  • Cameras are capturing faces, screens, other workstations or conversations that the task does not need.
  • The robot vendor or integrator contract is silent or restrictive about the business's rights to controller logs.
  • Corrections are clustering around a hazard, such as people entering the cell during operation; that is a safety review before it is a dataset.
  • Sites in more than one state or country are pooling data under one notice.

Offering correction records from your own floor

If your business already runs robots with human oversight, the first question is whether your records meet a specification like the one above, and the second is whether you have the rights to license them. Both are answerable before any data moves.

SourceX works with businesses that want to license operational recordings like these, covering rights review, contracting, preparation and delivery. A short company assessment comes first and needs no files, and long robot-cell recordings can stay on your own storage or be shipped on encrypted drives instead of being hosted by SourceX.

Frequently asked questions

What is human correction data in physical AI?

It is a record of a person taking over from a robot during a task, fixing what went wrong and handing control back, captured with synchronized video, robot state and actions. Each frame is tagged with who was in control, and the event carries a reason and an outcome.

Is a video of a worker fixing a robot enough?

Usually not. Without the robot's state and actions on the same clock and a logged signal of when control changed hands, a model cannot learn what the robot did wrong or what the worker did differently. Video alone suits demonstrations of human technique, not corrections.

Should e-stops be included in a correction dataset?

Keep them out of imitation data. An e-stop is a safety event that a safety reviewer should examine; the robot's behavior after it is not a correction to copy. It can sit in a separate, labeled failure log if it is useful for evaluation.

Do workers need to be notified before correction events are recorded?

Many states set rules for workplace electronic monitoring, and they differ in scope; Connecticut, for example, requires prior written notice of the types of monitoring and, from October 1, 2026, the locations. Counsel should confirm the rules for each site, and notice to workers is separate from the business's right to license the data.

How long should the pre-trigger window be?

Long enough to show the robot's error developing, which depends on the task speed. Set it per task in the specification, record it on every event and check a sample to make sure the cause is visible before the worker steps in.

Sources

  • LeRobot's human-in-the-loop documentation describes recording human recovery and correction alongside autonomous segments to address distribution shift, a pause, takeover and return-to-policy loop, and notes that no frames are recorded during a pause. Source
  • LeRobot's rollout configuration tags intervention frames with an intervention flag set to true. Source
  • The RLDS format marks the first and last step of each episode and distinguishes terminal steps from truncated episodes. Source
  • rosbag2 records timestamped messages, distinguishes reception and publication timestamps, and with simulated time stamps messages from the clock topic. Source
  • HG-DAgger lets a human expert monitor a learning policy, take control when needed and return control afterward. Source
  • Connecticut General Statutes 31-48d requires employers engaged in electronic monitoring to give prior written notice of the types of monitoring, and a conspicuous posting counts as that notice. Source
  • Connecticut Public Act 26-73, effective October 1, 2026, requires the monitoring notice to include the specific workplace locations where monitoring may occur, with exceptions such as security and employee safety monitoring. Source
  • New York Civil Rights Law 52-c requires employers that monitor telephone, email or internet use to give prior written notice on hiring, acknowledged by the employee. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies