Short answer
A clip-level metadata schema records, for every demonstration clip, where it came from, who and what appears in it, how it was captured, which labels exist and which rights apply. Typical field groups are identity and lineage, site and context, demonstrator, task, capture and calibration, quality flags, annotation status and permissions. Fix names, types and allowed values before collection so suppliers can fill them and you can filter and split.
Why clip-level metadata decides what you can do
Video without per-clip metadata can be watched but not managed. You cannot hold out a site if clips do not carry a site ID, cannot match a camera if the device is unknown, and cannot honor a consent withdrawal if you cannot find that person's clips. Metadata also drives the coverage reports that show whether a collection met its plan.
Dataset-level documentation is a separate artifact. A datasheet for datasets or data card describes the whole collection, its purpose and its known gaps; adapting that template for physical data is covered in datasheets and data cards for robotics datasets. This page is the row-level complement: the fields every clip must carry.
What published datasets record per episode
Robot datasets show the value of recording context with every episode. DROID stores camera calibration, depth and a language instruction with each episode (Khazatsky et al., 2024). RLDS defines episodes as sequences of steps holding observation, action, reward, discount and is_first, is_last and is_terminal flags (Ramos et al., 2021). LeRobot datasets pair Parquet tables with MP4 video per camera and describe the dataset in a meta/info.json file (features, frame rate, codebase version, path templates) plus task and episode metadata files (Hugging Face). Ego-Exo4D releases camera poses alongside its multi-camera video (Grauman et al., 2023).
Human demonstration clips have no robot actions or rewards, so their schema keeps the episode-level ideas (identity, context, calibration, task, language) and replaces per-step action fields with video timestamps and labels. Do not force human video into an action schema with placeholder zeros; a policy trained on those rows is being taught to output no motion.
Field groups
| Group | Example fields | Why buyers need it |
|---|---|---|
| Identity and lineage | clip_id, source_file_hash, parent_recording_id, delivery_batch, processing_version | Deduplication, traceability, re-delivery |
| Site and context | site_id, station_id, setting, lighting, shift | Splits, coverage reports, confound checks |
| Demonstrator | operator_id (pseudonymous), experience_band, handedness, glove_type | Person-level splits, proficiency analysis |
| Task | task_code, variant, step_vocabulary_version, procedure_ref, outcome | Filtering, success labels |
| Capture | device_model, unit_id, mount, resolution, frame_rate, shutter, codec, start_time_utc, duration_s | Sensor matching, quality control |
| Calibration and sync | intrinsics_ref, extrinsics_ref, clock_source, sync_offset_ms | Geometry and multi-stream alignment |
| Quality | hands_visible_share, blur_flag, dropped_frames, qc_status | Filtering and acceptance |
| Annotation status | labels_present, guideline_versions, double_annotated | Knowing what exists for each clip |
| Rights and privacy | consent_record_ref, permitted_uses, redaction_version, retention_until, withdrawal_status | License compliance and deletions |
Illustrative schema
# Illustrative only — field names, types and enumerations a buyer adapts
schema: demo_clip_metadata
version: 0.4.0
fields:
clip_id: {type: string, required: true, pattern: "clip_[a-z0-9_]+"}
source_file_hash: {type: string, required: true, note: "hash of the delivered file"}
parent_recording_id: {type: string, required: true}
processing_version: {type: string, required: true, note: "transcode + redaction pipeline"}
site_id: {type: string, required: true, example: site_017}
station_id: {type: string, required: false}
setting: {type: enum, values: [staged, off_shift, live], required: true}
lighting: {type: enum, values: [day, night, task_light, mixed, unknown]}
operator_id: {type: string, required: true, note: "pseudonymous, stable across deliveries"}
experience_band: {type: enum, values: [trainee, qualified, expert, unknown]}
glove_type: {type: enum, values: [none, nitrile, cut_resistant, leather, other, unknown]}
task_code: {type: string, required: true, example: KIT-04}
outcome: {type: enum, values: [success, recovered, failed, abandoned, unknown]}
device_model: {type: string, required: true}
unit_id: {type: string, required: true}
mount: {type: enum, values: [head, chest, wrist, fixed, handheld]}
frame_rate: {type: number, unit: fps, required: true}
start_time_utc: {type: datetime, precision: minute, required: true}
duration_s: {type: number, required: true}
intrinsics_ref: {type: string, required: false}
sync_offset_ms: {type: number, required: false}
hands_visible_share: {type: number, range: [0, 1]}
qc_status: {type: enum, values: [pass, fail, review]}
labels_present: {type: list, values: [steps, hoi, language, hand_pose_3d]}
consent_record_ref: {type: string, required: true, note: "points to a record, not a yes/no flag"}
permitted_uses: {type: list, values: [training, internal_eval, benchmarking]}
redaction_version: {type: string, required: true}
retention_until: {type: date, required: true}
withdrawal_status: {type: enum, values: [active, withdrawn_pending_deletion, deleted]}
nulls: {unknown: "value existed but was not recorded", not_applicable: "field does not apply"}
Controlled vocabularies and IDs
- Stable pseudonyms. Operator and site IDs must stay the same across deliveries, or splits and withdrawals break. The key that maps pseudonyms to people stays with the supplier; see pseudonymization.
- Versioned enumerations. Settings, glove types and outcomes come from fixed lists with a version number.
- Units in names. duration_s and sync_offset_ms leave no doubt about units.
- Time. UTC with an explicit format; round start times where minute precision is enough.
- Distinct nulls. Unknown, not applicable and not recorded mean different things and should be coded differently.
Storage and delivery
A single table for all clips is easiest to query; Parquet is a column-oriented format with compression and encoding schemes suited to that (Apache Software Foundation). For training pipelines that stream data, WebDataset groups files that share a basename into one sample inside tar shards (WebDataset project), so a clip's video and its JSON sidecar travel together. Ask for both views if your teams work differently. Containers, codecs and annotation files are covered in delivery formats for video datasets.
Validation on delivery
- Schema checks. Required fields present, types correct, enumerations respected.
- Cross-field checks. Frame rate and duration in metadata match the file; held-out operators never appear in training batches; outcome is set wherever step labels exist.
- Referential integrity. Every consent_record_ref resolves to an entry in the consent register extract, and every calibration reference resolves to a file.
- Lineage. source_file_hash matches the delivered file.
These checks slot into the wider acceptance criteria for demonstration video, and the IDs they rely on are what make splitting demonstration data by site and operator possible.
Privacy inside metadata
Metadata can identify people even when faces are redacted: an operator ID, a site, a shift and a precise start time can be enough. Keep only the precision you need, keep the mapping key with the supplier and assess re-identification risk on the metadata itself. Large egocentric datasets document consent and de-identification procedures alongside the data; Ego4D, for example, describes consenting participants and de-identification (Grauman et al., 2021). Point to those records from each clip rather than summarizing them in a flag. How records should follow data from capture to delivery is covered in provenance and chain of custody.
Limitations
- Metadata cannot recover information that was never captured; archives often lack device models, operator IDs or per-clip consent records.
- A consent flag is not evidence. Reference the underlying record.
- Very long schemas get filled inconsistently. Mark fields as required or optional and keep the required set short.
- Metadata does not change what the footage contains: human video still has no robot actions or forces.
Questions to ask suppliers
- Which fields can you fill from existing records, and which need manual entry?
- Are operator and site IDs pseudonymous and stable across deliveries?
- Where are consent and permission records held, and how will each clip reference them?
- Can you report device model, unit and settings per clip, or only for the archive as a whole?
- How are processing versions, such as redaction and transcoding, tracked per clip?
- In what format will metadata be delivered, and will a machine-readable schema file accompany it?
- How will withdrawals and deletions be reflected in later deliveries?
- Which fields are unknown for older footage, and how will that be coded?
Discuss your requirements
If you have a draft schema or need one filled for footage you want to license, send SourceX the fields, the asset type and the permitted uses. SourceX reviews the requirement, checks the data and the supplier's licensing permissions, and follows up about potential sources; a request does not guarantee a match. Other capture and delivery topics are on the workplace task demonstration hub and the humanoid training data guide.
Explore sourcing and licensing options with SourceX
Sources
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode calibration, depth and language.
- RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning — episode and step structure for robot logs.
- LeRobot (Hugging Face repository) — dataset format with a schema file plus task and episode metadata.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — camera poses released with multi-camera video.
- Ego4D: Around the World in 3,000 Hours of Egocentric Video — documented consent and de-identification.
- Apache Parquet — columnar storage for metadata tables.
- WebDataset — sharded samples that keep video and sidecar files together.