Skip to content

Systems and records

Robotics video action outcome alignment: linking each movement to its result

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

To link a hand movement to its result, put the video, tool signals and business records on one clock, store each action as a segment with start and stop times, and point it to a dated outcome record from a named system, such as a torque log or inspection result. A validator should reject any link where the outcome predates the action or names another unit.

Video and sensors

Robotics video action

This page covers robotics video action outcome alignment: linking each movement to its result.Illustrative

Key takeaways

  • Most results of physical work are measured by tools and systems, not visible on camera, so outcomes must come from those records.
  • Every stream needs a named clock and a measured offset; arrival time is not capture time.
  • Store each action as a segment with start and stop frames and link it to an outcome that names the same unit.
  • Reject any link where the outcome predates the action or the unit does not match, and publish the validator results.
  • Failures, corrections and consent records belong in the dataset as first-class fields.

You link it with three things the footage cannot supply on its own: a shared clock, an action segment with explicit boundaries, and an outcome record that names the unit of work it describes. A worker tightening a fastener, seating a connector or folding a garment produces a result that is often invisible on camera. The torque value lives in the tool controller, the pass or fail lives in an inspection system, and the rework order lives in the manufacturing execution system. Robotics video action outcome alignment is the work of joining those records to the right seconds of video.

Robot learning formats already separate actions from results. The RLDS format stores episodes as sequences of steps, each with an observation, an action and optionally a reward, plus flags for the first, last and terminal step. When an episode ends on a step that is not terminal, RLDS treats it as truncated. Workplace video destined for those pipelines needs the same distinctions made explicit: where the action ended, whether the task finished or was cut off, and what the result was.

The rest of this page gives the minimum fields of an action and outcome synchronization schema, the purpose of each, a valid illustrative record, a failing one, the validator checks that catch the difference, and the consent and coverage questions a company should settle before filming.

Why the camera rarely sees the result

Most outcomes in physical work are measured, not seen. A fastener looks the same at the right torque and at half of it. A weld bead can look clean and fail a later test. A pick looks successful until the scan at the packing station shows the wrong item. If the dataset labels outcomes from the video alone, it will mark many failed attempts as successes, and the model will learn the wrong lesson from them.

The outcome therefore has to come from the system that measured it, carry its own timestamp, and identify the unit, such as a serial number, work order, tote or order line. When no system measured it, label the outcome as visually judged and keep it apart from measured outcomes, because the two have very different reliability.

Delayed outcomes need their own handling. An inspection may happen an hour after the action, and a warranty return months later. Record the outcome time and the action time separately, and record which outcome sources were checked and when, so that a missing failure is not mistaken for a success.

Putting camera, tools and records on one clock

Every stream needs a timestamp from a known clock, and the dataset must record which clock and how the offsets were measured. Logging formats draw this line carefully. The MCAP specification gives each message a log time, when it was recorded, and a publish time, when it was published, and requires publish time to fall back to the log time when unknown. Treat the camera frame time, the tool controller time and the business record time the same way: keep the original value, and store the corrected value alongside it.

Matching rules should be stated, not implied. The ROS 2 message_filters library offers three synchronization policies: exact time, exact within an epsilon tolerance, and an adaptive approximate match. Its Python approximate synchronizer can accept messages without a header timestamp by substituting the current ROS time, which is arrival time rather than capture time. A dataset that silently does the equivalent will place outcomes late by however long the record took to arrive.

Store frame-accurate times for actions. LeRobot's dataset format keeps states, actions and timestamps in Parquet tables and camera frames in MP4 files, with per-episode offsets in metadata, and its documentation notes that exact event matching has no tolerance window, so event rows must carry the frame timestamps from the tabular data. A simple practice for workplace capture is a visible and logged sync event at the start of each session, such as a scan of a barcode that both the camera and the scanner record, so the offset between clocks can be measured rather than assumed.

The action and outcome synchronization schema

These are the minimum fields for one action segment linked to one outcome. A task episode holds several segments, and one segment can link to more than one outcome when several systems measured the result. Field names are suggestions; what matters is that every field is present and checked.

The action and outcome synchronization schema
FieldPurposeFormatRequired check
episode_id and task_idTies the segment to one recorded task attempt and its task definitionOpaque IDs; task_id maps to a written task descriptionBoth resolve; task description versioned
segment_start and segment_stopBoundaries of the hand movement on the shared clockTimestamps with clock_id, plus start and stop frame numbersStop after start; frames match timestamps at the stated frame rate
action_labelWhat the hand did, as a verb and objectClosed vocabulary, such as seat connector or torque fastenerValue in vocabulary; free text kept in a separate note field
actor_roleWho performed the actionWorker, robot, teleoperator, or a handoff between themOne value per segment; handoffs split into two segments
unit_refThe physical unit the action changedSerial number, work order line, tote or order lineSame unit_ref appears on the outcome record
outcome_ref and outcome_sourceThe record of the result and the system that holds itRecord ID plus system name, such as torque controller or inspection stationRecord exists in an export taken for this dataset
outcome_time and outcome_valueWhen and what was measuredTimestamp on the shared clock; value with units or pass, fail, reworkOutcome time not before segment_stop
outcome_basisHow the result is knownMeasured, inspected, visually judged, or not yet knownVisually judged and not yet known never merged with measured
interventionWhether someone corrected the attemptNone, recovery, correction, or abandonIntervention segments carry their own boundaries
consent_ref and visibilityConsent record and what parts of people are visibleConsent record ID; hands only, upper body, face visibleConsent record covers the date and the use

A valid record and a failing one (illustrative)

Both records are invented for illustration and do not describe any company's data. In the valid record, a worker at an assembly station seats a wiring connector and then torques two fasteners on the same housing. The episode holds three segments. The torque controller logs two results against the housing's serial number, each a few seconds after its fastening segment ends; both pass. The connector has no measured result at the station, so its outcome_basis is inspected, linked to the end-of-line inspection record for the same serial number, which logged a pass later the same shift.

The failing record looks plausible at a glance. Its fastening segment links to a torque result whose timestamp is earlier than the segment start, and the result carries a different serial number. Investigation shows the torque log was matched by nearest time across two adjacent stations, and the station clock ran behind the camera clock. Three checks catch it: outcome time before segment stop, unit_ref mismatch, and a clock offset recorded as unmeasured.

The exception worth keeping: when a worker re-torques a fastener after a fail, the result log holds two outcomes for the same unit and fastener. The schema handles this as two segments, the second carrying intervention correction, each linked to its own outcome, so the failed first attempt stays in the data as a failure rather than being overwritten by the later pass.

Validator checks to run before any footage leaves the building

Run these checks on every record and publish the failure counts with the dataset. They are mechanical and reproducible, which is the point: a buyer should be able to rerun them on delivery and get the same result.

At a glance

Validator checks to run before any footage leaves the building

Decide in advance what happens to a record that fails. A clock or unit failure means the link is wrong, so remove the outcome link and keep the segment with outcome_basis not yet known; do not guess a replacement match by nearest time. A consent failure means the episode leaves the dataset entirely. Keep a log of every removal with the check that triggered it, because a buyer will ask why a station or a shift has fewer episodes than expected.

  • Clock: every stream names a clock_id, and every clock pair has a measured offset with the method recorded.
  • Order: segment_start precedes segment_stop, and outcome_time is not earlier than segment_stop.
  • Frames: start and stop frame numbers match the timestamps at the declared frame rate.
  • Unit: unit_ref on the segment equals unit_ref on the linked outcome record.
  • Source: every outcome_ref resolves in a dated export from the named system.
  • Basis: outcome_basis is present, and visually judged outcomes are counted separately in every summary.
  • Coverage: each task has failed and corrected attempts as well as successes, or the dataset card says it does not.
  • Consent: every episode resolves to a consent record valid on the recording date for the stated use.

Does the footage show variation, failures and recovery?

A set of clean, successful repetitions teaches a model how the task looks when nothing goes wrong, which is the case it needs least. LeRobot's human-in-the-loop guide makes the same argument for robot policies: training only on successful demonstrations lets small errors compound into unfamiliar states, and recording human recovery and correction alongside autonomous segments teaches a policy to handle failures. Workplace video should capture the equivalent: the dropped part, the cross-threaded fastener, the second attempt.

Label action boundaries separately from any spoken or written description. The EPIC-KITCHENS-100 annotations keep the time a participant narrated an action apart from the start and stop times of the action segment itself, and note that narration timestamps alone support only weakly supervised methods. A worker's voice note or a supervisor's log entry is useful context, but its timestamp is not the action's boundary.

Record operating conditions with each episode: shift, station, tool model, part variant, lighting and anything unusual. Without them a buyer cannot tell whether a gap in failures reflects skill or simply a narrow slice of conditions.

How linked records map into robot learning formats

Keep the schema as the source of truth and convert it into whatever format a buyer trains on, rather than designing the capture around one format. The mapping is mostly direct. The episode_id and task definition become episode metadata; RLDS proposes optional episode fields such as an episode ID and an invalid flag for incomplete episodes, and LeRobot maps natural-language task descriptions to integer IDs in a tasks file. Action segments become ranges of steps or frames, and outcome fields become episode or segment metadata rather than per-frame values.

Two decisions need to be written down during conversion. First, whether an episode that ended without a measured outcome is marked as truncated or excluded; RLDS distinguishes a terminal last step from a truncated one, and the conversion should preserve that difference instead of defaulting everything to complete. Second, how outcome_basis travels: a measured pass and a visually judged pass should not collapse into the same success flag. Ship the conversion script and the validator together so the buyer can trace any training example back to the original video seconds and the original outcome record.

Decide the task boundaries and the camera framing before recording, and get written consent from the people who will appear. Filming hands at a fixed station is a different privacy question from head-mounted cameras that also capture coworkers, screens and badges. Record in consent_ref and visibility exactly what each episode shows, and exclude footage of people who did not consent rather than trying to blur them afterward.

Employee recording rules vary by state and by whether a workforce is represented, and a camera that also captures audio raises separate consent questions. Access to the footage, the workers' privacy permission, ownership of the recordings and the contractual right to license them are four different things; holding the video does not settle the other three. Counsel and a workplace privacy reviewer look at the specific plan before anything is filmed for licensing.

Offering task video that comes with outcome records

A manufacturer, warehouse or service business that already logs torque results, inspection outcomes or scan events has the harder half of this dataset. Video with linked, measured outcomes is far more useful to robotics and physical AI teams than footage alone, and the records often exist already in quality and execution systems.

Through SourceX, a company with linked task video can take it from rights review and contracting to preparation, delivery and payment as a licensed AI dataset. A company can find out whether it qualifies through a short assessment that needs no uploads. It approves scope and terms and signs before anything is delivered, and footage too big to move over the network can remain on the company's servers or travel by encrypted drive.

Frequently asked questions

What should an action and outcome synchronization schema include?

At minimum: episode and task IDs, action start and stop on a named clock with frame numbers, an action label from a closed vocabulary, the actor, the unit the action changed, the outcome record and its source system, the outcome time and value, how the outcome is known, any intervention, and a consent reference. Each field needs a check a validator can run.

What operating variation or recovery behavior is usually missing from the footage?

Failed attempts, corrections and abandoned tasks are the usual gaps, along with unusual conditions such as a different part variant, a worn tool or a night shift. Many collections also record only the best operators. A dataset card should state which of these are present and which are not.

What makes a linked record invalid?

An outcome timestamp earlier than the end of the action, an outcome that names a different unit, a clock offset that was assumed rather than measured, or an outcome inferred from video but recorded as measured. Each of these can be caught mechanically before delivery.

Do I need a robot to produce robotics training video?

No. Video of people doing physical tasks, linked to measured results, is useful to robotics and physical AI teams as long as the recording conditions, task definitions and outcomes are documented. Robot formats such as RLDS and LeRobot can then hold the converted data.

Can existing CCTV footage be linked to outcomes?

Sometimes, if the cameras keep reliable timestamps and the views show the work clearly enough to mark action boundaries. Security footage was usually collected under a different notice and purpose, so consent and the right to license it need fresh review before any reuse.

Sources

  • RLDS stores episodes as steps with observation, action and optional reward, flagged as first, last or terminal; an episode ending on a non-terminal step is truncated. Source
  • LeRobotDataset v3 keeps states, actions and timestamps in Parquet and camera frames in MP4, with per-episode offsets in metadata. Source
  • LeRobot event matching has no tolerance window, so event rows must carry frame timestamps from the tabular data. Source
  • LeRobot's human-in-the-loop guide records human recovery and correction alongside autonomous segments because policies trained only on successful demonstrations compound small errors. Source
  • EPIC-KITCHENS-100 annotations record narration timestamps separately from action segment start and stop times. Source
  • MCAP messages carry a log time, when recorded, and a publish time, which must be set to the log time when not available. Source
  • ROS 2 message_filters synchronizes channels by header timestamps under exact, epsilon-tolerance or approximate policies; the Python approximate synchronizer can substitute current ROS time for a missing header stamp. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies