Delivery Formats for Video Datasets: Codecs, Containers and Annotation Files

The buyer's problem

Video datasets arrive in whatever form the supplier's systems produce: proprietary NVR exports, MP4 files with variable frame rates, annotation spreadsheets, labels keyed by filenames that the video folder does not match. Each mismatch costs engineering time, and some cannot be fixed afterward. A variable-frame-rate file with dropped timestamps may make step times unrecoverable. Over-compressed video may lose the fingertips you wanted to track.

Delivery format belongs in the license or statement of work rather than being left to goodwill. This page lists the choices and what to write down. For robot logs, joint states and action streams, see robot learning data formats; this page is about video and its annotations. Contract-level terms on use, redistribution and deletion are in licensing terms for robotics training data. This is not legal advice.

Video: choices that affect training

Three points deserve emphasis. First, ask for the original files when feasible and a transcoded training copy, with both traceable to each other, because transcoding is lossy and irreversible. Second, require constant frame rate or an explicit per-frame timestamp table. Third, check that tools can seek to a frame in your pipeline; some long-GOP files decode slowly for random access.

Annotation files

Annotations should be separate from video, in open formats, and linked by stable IDs.

• Masks, boxes, categories. COCO-style JSON is a widely adopted convention for instance segmentation; COCO itself collected per-instance segmentation from crowd workers for about 2.5 million labeled instances across 91 object categories (Lin et al., 2014). Specify run-length-encoded masks or polygons and the coordinate convention.
• Temporal segments and per-frame signals. Columnar tables work well. Apache Parquet is an open column-oriented format with compression and encoding schemes (Apache Parquet). One row per segment or per frame, with typed columns, avoids ambiguity that spreadsheets produce.
• Hand pose, body pose, tracks. Tables with one row per frame per entity, plus a schema listing joint names, units and coordinate frame.
• Language. One row per caption or instruction with span start and end, speaker or annotator ID, and language code.

Name the schema version in the file. If labels change after a rubric update, require a version bump and a change log. See demonstration clip metadata schema for clip-level fields.

Packaging for training

For very large collections, a sharded archive reduces small-file overhead. WebDataset stores samples in tar shards, groups files sharing a basename into one sample, and streams from local disks or cloud object stores into PyTorch, TensorFlow or JAX (WebDataset project). That is a buyer-side convenience; request it only if your pipeline will use it, since you can build shards from a plain delivery. Plain directories with a manifest suit smaller sets and audits.

When video sits alongside other timestamped streams, such as telemetry from a robot rig or handheld device, a log container is the usual choice. MCAP is an open-source container for multimodal log data that holds multiple timestamped channels, is serialization-agnostic and is the default ROS 2 bag storage since Iron (MCAP). Plain workplace video rarely needs it, but a collection with synchronized sensors might. Alignment with business event logs is covered in aligning demonstration video with workplace system logs.

What the format cannot do

A neat format does not make data sufficient. It cannot add actions or forces that were never recorded, repair a bad timestamp, or establish permission to use the video. A manifest with checksums proves what arrived, not that the content is allowed or good. Conversion between formats can also drop information, such as burned-in overlays, rotation flags or non-standard color profiles.

Illustrative delivery specification

# Illustrative only - values are examples a buyer would set
delivery: workplace_video_v0
video:
  original: native files retained, hash recorded
  training_copy: mp4 container, H.264 high profile, constant 30 fps, 1920x1080, yuv420p, keyframe every 30 frames
  audio: stripped from training copy; originals retained under separate terms
  timestamps: per-clip parquet with frame_index, pts_ms, wall_clock_utc, clock_source
layout:
  clips/{site_id}/{camera_id}/{clip_id}.mp4
  annotations/{label_type}/v{schema_version}/*.parquet
  masks/coco_instances_v{n}.json
  manifest.parquet: clip_id, path, sha256, bytes, duration_s, fps, width, height, site_id
  datacard.md: collection process, consent basis, known issues, schema versions
transfer: encrypted, checksummed; manifest signed by supplier; staged delivery of 3 batches
corrections: versioned; superseded files kept and marked

Acceptance tests for a delivery

1. Every manifest path exists, and every file on disk is in the manifest.
2. SHA-256 checksums match.
3. A decode test runs on every file, with zero failures, and frame counts match the declared duration times frame rate within a tolerance.
4. Timestamps are monotonic; there are no negative or duplicate presentation times.
5. Annotation IDs resolve to existing clips and frames; enumerations validate against the schema.
6. A random sample is visually checked for color, rotation and overlays.
7. Documentation is present and its stated counts match the files. A data card or similar summary is a sound template (Pushkarna et al., 2022).

More on checking deliveries is in acceptance testing for deliveries and demonstration video acceptance criteria. Storage and transfer sizing is in storage and transfer planning for video.

Common delivery faults to prevent

Most delivery problems are avoidable with a stated rule. Clips sometimes arrive with rotation flags that players honor and training loaders ignore, so frames load sideways. Variable frame rate files shift timestamps in frame-index-based loaders. Burned-in timestamps or watermarks leak into model input. Annotation files reference clip names that were later changed during redaction or re-encoding. Audio tracks remain in files where the contract said audio would be removed, creating a privacy problem. Add a clause that the supplier will run your validator, or an agreed equivalent, before shipping, and that nonconforming files are replaced at the supplier's cost. Keep a record of every version delivered so a model trained on one version can be traced to it.

Questions to ask suppliers

1. What formats do you hold the originals in, and can you provide them?
2. Is frame rate constant, and where do timestamps come from?
3. What compression settings were used, and has the footage been transcoded more than once?
4. Do annotation files carry a schema version and change log?
5. Can labels be exported as COCO-style JSON or Parquet tables?
6. How will you transfer and verify the delivery?
7. How are corrections and relabeled files versioned?
8. Which files contain personal data, such as audio, and how are they segregated?

Discuss your requirements

If you are specifying video and annotation delivery, describe the formats, volume and intended use to SourceX. SourceX coordinates delivery as part of its role in sourcing, rights review and licensing; it reviews the requirement and follows up about potential sources, and a request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub.

Explore sourcing and licensing options with SourceX

Sources

• WebDataset — tar-shard packaging and streaming.
• Microsoft COCO: Common Objects in Context — instance annotation convention.
• MCAP — container for multimodal timestamped data.
• Apache Parquet — columnar format for annotation tables.
• Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — dataset documentation template.