Short answer
Sensor synchronization means every stream in an episode, camera frames, joint states, actions, force and IMU, can be placed on one common time base with a known error. Specify the clock source, the maximum offset between streams, how timestamps are stored, and the test that proves it. IEEE 1588 Precision Time Protocol is a common network mechanism; file formats like MCAP preserve the result.
The buyer's problem
A policy learns from pairs of observation and action. If the camera frame lags the joint state by tens of milliseconds and nobody records it, the model learns a relationship that does not exist on the robot. Misalignment rarely shows up as an error message. It can show up as a policy that looks fine in training replay and behaves differently on hardware.
Synchronization problems have three common causes: different clocks on different machines, buffering and driver latency that make a timestamp reflect arrival time rather than exposure time, and variable frame rates in compressed video. Each can be specified against and tested, but only if the dataset keeps enough timing information.
Where timestamps come from
| Stream | Typical timestamp source | Common issue |
|---|---|---|
| Camera frames | Exposure start, driver receive time, or encoder time | Receive time lags exposure by an unknown amount |
| Joint states and actions | Controller clock | Controller clock differs from the computer clock |
| Force/torque | Sensor clock or bus receive time | Sampling rate far above the camera rate |
| IMU | Sensor clock | Offset relative to camera needs estimating |
| Audio | Sample counter | Drift over long sessions |
Ask for the timestamp definition per stream, not one general statement.
Clock sources and PTP
IEEE 1588-2019 defines the Precision Time Protocol for synchronizing clocks across networked systems, including Ethernet, with sub-microsecond accuracy possible using minimal resources, and supports customization through profiles (IEEE, 2019). In practice, a collector using PTP would have devices synchronized to a grandmaster clock, with hardware timestamping on the network interfaces. The achieved accuracy depends on hardware and network design, so ask for measured offsets, not the standard's headline capability.
Alternatives include hardware trigger lines that fire all cameras at once, a shared sync pulse recorded on every device, or software estimation of offsets after the fact. Each has a different error profile. A hardware trigger gives frame-level alignment between cameras but does nothing for a force sensor on a different bus unless that signal is also recorded. A buyer's requirement should name the tolerance and leave the method open, then verify.
Storing timing: containers and logs
MCAP is an open-source container for multimodal log data: it holds multiple channels of timestamped, pre-serialized messages, is serialization-agnostic (ROS 1, ROS 2, Protobuf, JSON, FlatBuffers and others), supports optional indexes and compression, and is the default ROS 2 bag storage since Iron (Foxglove, MCAP). The ROS 2 tooling records topic data with ros2 bag record and can replay it, with mcap as an example storage ID (Open Robotics, rosbag2 tutorial).
A container preserves timestamps; it does not make them correct. Many recorded messages carry both a header stamp set by the publisher and a log time set at recording. Require the dataset to state which is authoritative and keep both. More on layouts is in robot learning data formats.
What synchronization cannot fix
Good synchronization cannot add missing signals. Human workplace video has no joint states to align, and aligning video with system logs, such as machine events, is a different, coarser problem covered in aligning video with system logs. It also cannot repair a camera whose exposure timing is unknown or a video encoded with variable frame rate and no per-frame timestamps. Buyers who receive only compressed MP4 files lose frame-level timing unless the supplier delivers timestamps separately. Control frequency interacts with sync: see action space and control frequency.
Illustrative synchronization specification
# Illustrative - example values only, not recommended tolerances; buyer sets every value
sync_spec:
reference_clock: ptp_grandmaster_01
timestamp_definition:
camera: exposure_start
joint_state: controller_sample_time
wrench: sensor_sample_time
max_offset_ms:
camera_to_camera: 5
camera_to_joint_state: 10
wrench_to_joint_state: 5
storage: mcap
keep_both_stamps: [header_stamp, log_time]
measurement_evidence: "per-session offset report"
drift_check: "start and end of every session"
dropped_frame_rule: "flag, do not interpolate silently"
Verification tests a buyer can run
- Flash or clap test. Record a visible and audible event, such as an LED flash driven by a timestamped signal, and compare detection times across streams.
- Motion cross-correlation. Correlate camera-derived motion with joint velocity to estimate offset; look for a stable lag across episodes.
- Frame interval statistics. Compute the distribution of inter-frame timestamps per camera and flag gaps or jitter above your limit.
- Session start-to-end drift. Compare offsets at the start and end of long sessions.
- Cross-session stability. Check whether offsets differ between sessions, collectors or sites.
Folding these into acceptance testing for deliveries makes the requirement enforceable. Camera-IMU time offsets are often estimated during calibration; see calibration requirements.
Common timing failures to look for
- Constant lag. Camera frames systematically trail joint states by a fixed amount. This is estimable and correctable if it is stable and documented.
- Variable lag. Latency changes with load, which cannot be corrected by a single offset.
- Frame duplication. The driver repeats the last frame when a new one is late, so apparent frame count looks right while content is stale.
- Rate mismatch. A 30 Hz camera, a 500 Hz force sensor and a 15 Hz control loop need a documented resampling rule; interpolating force to camera times is a choice that affects contact timing.
- Clock steps. An operating-system time update during a session creates jumps in timestamps.
- Concatenated sessions. Files merged from different machines with different clocks.
Each of these is visible in inter-frame interval plots or cross-correlation tests, so ask for those plots in the delivery report.
Sourcing and preparation
Synchronized multi-sensor data usually comes from robots, instrumented rigs or specialist robot-data providers, not from business archives. DROID is an example of a standardized rig: about 76,000 trajectories collected by 50 data collectors with the same hardware at 13 institutions, at 15 Hz control (Khazatsky et al., 2024). That describes one study's design, not a threshold for yours. Plan for resampling to a common rate, for handling dropped frames, and for recording a sync report with each session. Route choices are discussed in existing assets, new collection or specialist providers.
Questions to ask suppliers
- Which clock source do you use, and is there hardware timestamping?
- How is each stream's timestamp defined: exposure, receive or log time?
- What offsets did you measure between cameras and between cameras and joint states?
- How do you detect and report drift within a session?
- Do you deliver per-frame timestamps, or only video files?
- What container do you use, and do you keep both header and log stamps?
- How are dropped or duplicated frames handled and flagged?
- Is force/torque or IMU data recorded on the same clock, and at what rate?
- Can you provide a flash or clap test for a sample session?
- Were any streams resampled or interpolated before delivery?
Discuss your requirements
If synchronized multi-sensor recordings are central to your program, describe the streams and timing tolerances to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. Begin at the humanoid training data guide or the procurement hub.
Discuss your humanoid training-data requirements with SourceX
Sources
- IEEE 1588-2019 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems — PTP defined by IEEE 1588-2019 for clock synchronization in networked systems.
- MCAP — MCAP holds multiple channels of timestamped messages.
- Recording and playing back data (ROS 2 rosbag2 tutorial) — ros2 bag records topic data for replay, with mcap as an example storage ID.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — Example of a standardized rig replicated across institutions at 15 Hz control.