Robot Trajectory Dataset Specification: Observations, Actions and Episodes

Why the episode needs a specification

Two datasets with the same number of hours can be unlike each other once you open them. One logs joint targets, one end-effector deltas; one has a wrist camera, one does not; one stamps every frame from a shared clock, one infers time from a nominal frame rate. A buyer who writes down only "teleoperated episodes, 1080p" will receive whatever the collector's pipeline emits.

This page defines what each episode should carry. It applies to data from robots, teleoperation rigs, handheld capture devices or instrumented setups; business archives and ordinary workplace video hold none of these fields. How to run a collection is in teleoperation data collection requirements; what an action value means is in action spaces and control frequency; the container choice is in robot learning data formats.

The shape of an episode

Two widely used layouts show the structure. RLDS stores sequential decision data as episodes that each hold a dataset of steps, and a step carries an observation, an action, a reward, a discount and flags marking the first, last and terminal steps (Ramos et al., 2021). LeRobotDataset keeps tabular data in Parquet files, one MP4 video per camera, and a meta/info.json file that declares features, frame rate, codebase version and path templates, alongside normalization statistics, a task list and episode metadata (Hugging Face, n.d.). Either way, the dataset says what each column is, at what rate, and where the video lives. The requirements below hold whichever container you choose.

Illustrative field specification

DROID shows the per-episode metadata a multi-site program recorded: camera calibration, depth and a language instruction with each episode, collected at 15 Hz control (Khazatsky et al., 2024). Pooled corpora such as Open X-Embodiment, which combine more than 1 million trajectories from 22 embodiments contributed by 21 institutions (Open X-Embodiment Collaboration, 2023), are easier to use when each contributor describes episodes in compatible ways.

Illustrative episode record

# Illustrative only — synthetic identifiers and values; field names are examples to adapt
episode:
  episode_id: ep_000412
  dataset_version: v1.0
  task: {task_id: tote_unpacking, instruction: "Take the items out of the tote and put them on the shelf", success: true}
  embodiment: {robot: humanoid_X, revision: r3, controller: ctrl_5.1, description_file: humanoid_X_r3.urdf}
  operator_id: operator_A12
  site_id: site_017
  calibration_session: cal_0319
  control_rate_hz: 30                  # declared; measured rate reported in QA
  streams:
    cameras:
      head_left:  {res: [1280, 720], fps: 30, encoding: "H.264, declared quality"}
      wrist_right: {res: [640, 480], fps: 30}
    proprioception:
      joint_position: {dim: 14, unit: rad, order: [l_shoulder_1, "…", r_wrist_3]}
      gripper_state:  {dim: 2, unit: fraction_closed}
    action:
      vector: {dim: 16, space: ee_pose_delta + gripper_target, frame: robot_base, unit: [m, rad, fraction]}
      pairing: "action[t] is issued after observation[t]"
steps:                                  # one row per step
  - {t_ns: 1760000000000000000, is_first: true,  is_last: false, is_terminal: false}
  - {t_ns: 1760000000033333333, is_first: false, is_last: false, is_terminal: false}
  - "…"
  - {t_ns: 1760000009900000000, is_first: false, is_last: true,  is_terminal: true}

Observations: where specifications usually fail

• Camera identity. Name every camera and keep the set constant across episodes, or declare which episodes lack which streams. "Camera 2" meaning different mounts in different lots is a common defect.
• Alignment. State how video frames map to steps. A frame count that differs from the step count is acceptable only if timestamps allow a defined alignment.
• Compression. Video is lossy unless declared otherwise. Ask for the codec, quality setting and whether any re-encoding occurred.
• Proprioception. List joint names, order, units and sign conventions. Supply a robot description file; the URDF format, for example, describes a robot as links connected by joints, but represents only tree structures and rigid links (ROS, n.d.), so closed-chain mechanisms need another description.
• Missing streams. Force, tactile and depth should be present for every episode that claims them, with a flag where they are absent.

Actions: pairing, definition and statistics

Write the pairing convention into the specification: whether the action at step t is the command issued after observation t, or the target recorded before it. Include the commanded value and the measured state in the same step, so that you can check tracking error and derive other representations. The space, frame and units must be spelled out; the decisions are in action spaces and control frequency. If the delivery includes normalization statistics, ask which episodes they were computed from, and compute your own on training lots only so evaluation data does not leak into them; see training vs evaluation datasets.

Time and boundaries

Timestamps should come from one declared clock, per step and per frame, with the declared rate and the measured rate both reported. The IEEE 1588 Precision Time Protocol is one standard way to synchronize clocks across networked devices, with sub-microsecond accuracy (IEEE, 2019); the field-level requirements are in sensor synchronization requirements. Dropped frames should be flagged, not silently duplicated.

Define episode boundaries: where an episode starts, whether reset motion is included, and how a terminal step (task ended) differs from a cut-off (time limit reached). Long-horizon tasks may need sub-task labels within an episode.

Acceptance tests to attach

The general delivery process, including quarantine and sampling, is in acceptance testing for dataset deliveries.

What a specification cannot do

• Create fields that were never recorded. Actions cannot be added afterward to ordinary workplace video; inferred or latent actions are a modeling step, not a recorded signal.
• Guarantee meaning. A file can validate and still use the wrong units or a mislabeled frame. Replay and sampled review catch what schema checks miss.
• Make data portable across robots. Episodes describe one embodiment; use on another needs retargeting or a common representation.
• Capture quality of behavior. A schema does not record operator skill or whether the demonstration was a good one. Ask for operator IDs and outcome labels.
• Fix the volume. How many episodes you need depends on architecture, task, embodiment and intended use.

Questions to ask providers

1. Which fields from the table above does each episode contain, and which do some episodes lack?
2. How are actions defined: space, frame, units, absolute or delta, and what is the pairing with observations?
3. At what rate are actions, joint states and each camera logged, and what is the measured rate and jitter?
4. Is the commanded value logged alongside the measured state?
5. What calibration is delivered, per session or per episode?
6. How are dropped frames, sensor gaps and failed episodes represented?
7. Can you deliver a robot description file and a schema validator I can run?
8. What do the normalization statistics cover, and can I recompute them?
9. Which container will you use, and what is lost if I convert it?

Discuss your requirements

A written episode specification is worth sending with the request. Describe the robot data you need to SourceX and it will investigate whether US businesses hold suitable material; if no existing asset fits, SourceX can scope a commissioned collection. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also engage specialist capture partners, scoped case by case. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and nothing is contracted until a supplier agrees. For context, see the procurement hub, the humanoid training data guide and the general requirements specification template.

Discuss your humanoid training-data requirements with SourceX

Sources

• RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning — episode and step structure with first, last and terminal flags.
• LeRobot (Hugging Face repository) — Parquet, per-camera MP4 and schema file layout.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode calibration, depth, language and 15 Hz control.
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models — pooled corpus across 22 embodiments.
• URDF (Unified Robot Description Format) — robot description as links and joints (partial verification).
• IEEE 1588-2019 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems — Precision Time Protocol for clock synchronization.