Aligning Demonstration Video With Workplace System Logs

The buyer's problem

Footage shows a person packing a box. The warehouse management system recorded that order 88231 was picked at 14:02:17, packed at 14:03:40 and shipped at 14:09:05. Joined correctly, the log tells you where each task began and ended, which item it was, and whether it succeeded. That is cheaper and more consistent than annotators watching hours of video and guessing.

Joined incorrectly, it misleads. If the camera recorder's clock is 40 seconds off, every label lands on the wrong task, and the model learns noise that looks like signal. Alignment is a measurement problem, not an assumption. This page is a specification for doing it, including what the logs can and cannot add. For the labels derived from the join, see success and outcome labels and pick confirmation as labels.

What business events can supply

Logs give ground-truth timing at the events they record and nothing between. They contain no visual data and cannot tell you how a motion was done. They are strongest as boundary and outcome labels, and as a check on annotator labels.

Clocks: where alignment goes wrong

Every device has its own clock. Camera recorders often use internal time that has drifted since last set. Servers sync to network time and may differ from handheld scanners. Daylight-saving shifts and timezone configuration, such as logs in local time and video in UTC, introduce offsets of whole hours.

For tightly coupled sensors, the Precision Time Protocol defined in IEEE 1588-2019 synchronizes clocks in networked systems, with sub-microsecond accuracy achievable on suitable networks (IEEE, 2019). Ordinary business systems rarely reach that, and most workplace alignment needs only a second or better. What matters is knowing the achieved error, not assuming it.

Three practical techniques:

1. Sync events. Record a visible and logged event: scan a barcode on camera at the start and end of each session, or flash a screen showing the system clock. The pair gives offset and drift per camera.
2. Event matching. Identify a few unambiguous events, such as a conveyor start or door scan, in both video and logs and fit an offset.
3. Continuous cross-check. Compare event times with video-detected actions across the session and watch for the offset changing.

When video and other sensors share a recording, a timestamped multi-channel container keeps channels on one time base; MCAP holds multiple channels of timestamped messages and is serialization-agnostic (MCAP). It is optional for video-plus-logs but useful if telemetry is also collected.

Join keys

A timestamp alone is a weak key. Prefer a compound key: station or camera ID, operator pseudonym, order or work-order ID, item ID, and a time window. Where the log has an identifier that is also visible in the video (a tote label, a work-order sheet), use it as a verification anchor. Keep the mapping table, because later reviewers need to see how rows were matched, and carry the join fields in the clip record described in demonstration clip metadata schema.

Illustrative alignment specification

# Illustrative only - values are synthetic examples a buyer would set
alignment_spec: video_log_join_v0
clocks:
  video: recorder internal clock; offset measured per session against sync_event
  logs: WMS server clock, synced to site time source; timezone: America/Chicago, stored UTC
  sync_events: barcode scan on camera at session start and end (2 per session)
measured:
  offset_s: per camera per session
  drift_ms_per_hour: per camera per session
join:
  keys: [station_id, order_id, sku, time_window_s]
  tolerance_s: 2.0
  rule: nearest log event inside window; ties flagged "ambiguous"
outputs:
  clip_labels: task_start, task_end, order_id_hash, sku, qty, outcome
  alignment_error: median and p95 seconds on 100 manually verified events
  unmatched: counts of video tasks without log events, log events without video
privacy:
  identifiers: operator and customer IDs replaced by hashes; hash key not delivered
  excluded_fields: customer name, address, payment data
rights: log fields licensed separately from video; both listed in the manifest

Acceptance criteria

1. Offset and drift are reported per camera per session, with the method.
2. A manually verified sample of events (for example, a few hundred) gives alignment error as a median and 95th percentile.
3. Unmatched video tasks and unmatched log rows are counted and explained.
4. Join keys and the mapping table are delivered.
5. Log-derived labels are marked by source, so you can compare them with annotator labels.
6. Event timestamps are interpreted correctly: confirm whether each event marks the start of an action or its recording.

Limits and extra preparation

Logs miss everything not entered: failed attempts that were not recorded, a pick abandoned and restarted, a worker helping a colleague. They also reflect system rules; if the WMS auto-confirms at dock close, its timestamps will not be action times. Manual overrides, backdated corrections and batch confirmations occur. A supplier should disclose such practices.

Logs often hold personal and commercial data: operator IDs tied to performance metrics, customer orders, prices, supplier names. Hashing identifiers is a technical step; it does not settle who may license what. See confidential business information in footage and worker consent and identifiable people. Performance data about identifiable workers deserves particular care. This is not legal advice.

To segment continuous recordings with logs, see continuous shift recordings vs curated clips. Temporal label structure is covered in step segmentation annotation.

When logs and video disagree

Disagreement is information, not only error. A log that shows a pick at a time when the video shows the worker elsewhere may reveal a batch confirmation, a clock fault or a reassigned task. Build a review step: sample disagreements, classify the cause, and fix the alignment or flag the label as unreliable. Count how often logs lead the action, lag it, or occur with no matching motion. Over a few sessions this gives a distribution of lead and lag per system that you can use to correct timing, as long as the pattern is stable. If it is not, do not use log timestamps as boundary labels; use them only to identify what the task was and whether it completed, and place boundaries by annotation.

Questions to ask suppliers

1. Which clocks do cameras and systems use, and how are they synchronized?
2. Was there any sync event recorded in the footage?
3. Are log timestamps at the action, at confirmation or at batch close?
4. How often are records backdated or edited?
5. Which identifiers link a log row to a visible task?
6. Can you provide the data dictionary for each log table?
7. Which log fields contain personal or customer data, and who has authority to license them?
8. What share of video tasks has no matching log event?

Discuss your requirements

If your project would benefit from business events paired with footage, describe the systems, tasks and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates delivery; a request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub.

Discuss your humanoid training-data requirements with SourceX

Sources

• MCAP — multi-channel timestamped log container.
• IEEE 1588-2019 — Precision Time Protocol for clock synchronization.