Short answer
Judge a robotics dataset offer against ten gated checks: fit to your use, capture quality, robot-specific streams, labels, coverage, split design, provenance, permissions, format and handling, and documentation. Mark each as required or preferred before reading the offer, request the listed evidence in writing, and fail any offer that misses a required gate. Volume figures do not substitute for any of the ten.
How to use this checklist
Offers arrive as totals: hours of video, thousands of trajectories, terabytes. Totals say nothing about whether the data matches your embodiment, task or rights position. This page is the master checklist for judging an offer before you commit. It covers any modality; other pages specialize it for demonstration video, procedural text and defect image sets. Checks on a delivered lot belong to acceptance testing; this page works earlier, on claims and samples.
Write the gates down first, from your requirements specification. Mark each required or preferred. A required gate that fails ends the evaluation; do not trade it against a low price or a large total.
Gates about what the data is
| Gate | Pass looks like | Evidence to request |
|---|---|---|
| 1. Fit to use | The offer names the model use, task family, embodiment (or states there is none) and modalities, and says what the data cannot support | Written statement of intended and unsuitable uses; composition summary; list of excluded content |
| 2. Capture quality | Camera types, mounts, resolution, constant frame rate, real timestamps and intrinsics are declared and match the samples | Device list; probe output on sample files; calibration files |
| 3. Robot streams | If actions are claimed: named embodiment, joint and gripper state, commanded actions with a defined action space, control rate, calibration and a synchronization report. If not claimed, the offer says so | Schema; two raw episodes; kinematic description; measured vs declared rate |
| 4. Labels | Guideline version, annotator process, agreement and an error audit on a sample you did not choose | Guidelines; audit sheet; disagreement statistics |
| 5. Coverage | Counts by site, operator, task, object and condition, with failures and recoveries shown | Coverage table computed from metadata, not a marketing total |
| 6. Split design | IDs let you hold out by your axis; evaluation lots are separable | ID fields; split manifest; statement of prior sale or publication |
Gate 3 is where human video and robot data part ways. Human workplace video has no joint states or commanded actions, so it passes gate 3 only by not claiming them. Robot episodes must carry the fields in the robot trajectory dataset specification. For gate 5, Open X-Embodiment shows the kind of breakdown to ask for: it reports more than 1 million trajectories alongside 22 embodiments, 527 skills and 21 institutions (Open X-Embodiment Collaboration, 2023). A single total hides which of those dimensions your target is thin on. Gate 6 is explained in training vs evaluation datasets.
Gates about rights, handling and records
| Gate | Pass looks like | Evidence to request |
|---|---|---|
| 7. Provenance | Every item traces to a recorder, site, date and rights holder; the license chain from recorder to seller is documented | Item-level provenance extract; original license text for any public components |
| 8. Permissions | Notice or consent covers AI training and transfer to you; third-party and confidential content is identified | Pseudonymous consent register extract; exclusions list; redaction rules |
| 9. Format and handling | Files open with the declared tools; manifest and checksums exist; transfer, storage location and access controls are stated; a deletion path exists | Manifest; reader instructions; handling description |
| 10. Documentation | A datasheet or data card states motivation, composition, collection process, recommended uses and limitations | The document itself; known-issues list; change log |
Gates 7 and 8 have their own pages: provenance and chain of custody, worker consent and identifiable people, confidential business information in footage and biometric privacy and workplace video. Gate 9 continues in delivery and transfer of large datasets.
For gate 10, Gebru and colleagues propose that every dataset carry a datasheet documenting its motivation, composition, collection process and recommended uses (Gebru et al., 2018). Data Cards play a similar role as structured summaries of the facts stakeholders need across a dataset's lifecycle (Pushkarna et al., 2022). A supplier that has never written one will find several of the other gates hard to evidence; fields for robotics data are in datasheets for robotics datasets. If your organization wants a shared vocabulary for data quality, ISO/IEC 5259-1:2024 offers an overview, terminology and a framework for assessing and improving data quality across the data life cycle, and the 5259 series adds parts on measures, management, process and governance (ISO/IEC, 2024). The thresholds for your task remain yours to set.
Running the review
- Send the gates and the evidence list with your first request, so the supplier can answer in one pass.
- Ask for a sample you draw yourself, stratified by site, operator and device. See requesting and reviewing dataset samples.
- Check the sample against the offer: do the counts, fields and labels match what was claimed?
- Record each gate as pass, fail or evidence outstanding. Outstanding is not a pass.
- Decide, and keep the record. It becomes the baseline for acceptance testing.
Illustrative scorecard
# Illustrative only — synthetic offer and values; thresholds are examples the buyer sets
offer: OFFER_2026_011
use: co-training on tote handling, 2 arms
gates:
1_fit: {level: required, result: pass}
2_capture: {level: required, result: pass}
3_robot_streams: {level: required, result: not_claimed, note: "video only; actions must come elsewhere"}
4_labels: {level: preferred, result: pass, audit_sample: 150 clips drawn by buyer}
5_coverage: {level: required, result: pass_with_note, note: "no night-shift data"}
6_splits: {level: required, result: pass}
7_provenance: {level: required, result: evidence_outstanding, missing: "license chain for site_004"}
8_permissions: {level: required, result: fail, reason: "notice covers staff training only"}
9_handling: {level: required, result: pass}
10_documentation:{level: preferred, result: partial}
decision: do_not_proceed # gate 8 failed and is required
Red flags
- Only totals are offered; no breakdown by site, operator or condition.
- The sample is curated by the supplier, or no sample is available at all.
- Robot data is described without an embodiment, an action-space definition or a control rate.
- "Cleared for AI training" appears with no consent record behind it.
- Provenance is given as a single statement that "all data was collected in-house".
- Redaction is presented as having settled permission questions.
- A datasheet is promised for after purchase.
- The license cannot be shown before the data is.
What a checklist cannot tell you
- Whether the data will improve your model. Gates test the offer against your specification, not the specification against your results. A small pilot answers that question.
- Whether the supplier's records are true. A document request shows what the supplier asserts. Audit rights and independent sampling test it; see retention, deletion and audit rights.
- How much is enough. Requirements depend on architecture, task, embodiment and intended use, so none of the gates carries a universal minimum.
- Legal conclusions. Gates 7 and 8 produce evidence for counsel, not a view on compliance. This is not legal advice.
Questions to ask suppliers
- What was this data originally collected for, and by whom?
- Which embodiment, sensors and action space does it assume, or does it contain none of these?
- Can you report coverage by site, operator, task, object and condition from existing metadata?
- Who owns the rights at each link between the people recorded and the license you are offering?
- Which gates in this checklist can you evidence today, and which would need new work?
- Will you let me draw a review sample myself?
- What datasheet or data card exists now, and what does its limitations section say?
- What happens to items that fail a later check: replacement, removal or refund?
Discuss your requirements
Put the gates into the requirement you send through the buyers page: which are required, which evidence you will want and what the data must never contain. SourceX looks for US businesses that hold the data you describe, checks the data and the supplier's licensing permissions, agrees pricing and allowed uses in a license, and coordinates delivery and payment. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and nothing is contracted until a supplier agrees. Related reading: the procurement hub, the humanoid training data guide and the catalog's training-data due diligence checklist.
Discuss your humanoid training-data requirements with SourceX
Sources
- Datasheets for Datasets — datasheets documenting motivation, composition, collection process and recommended uses.
- Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset summaries across the lifecycle.
- ISO/IEC 5259-1:2024 Artificial intelligence - Data quality for analytics and machine learning (ML) - Part 1: Overview, terminology, and examples — data quality framework and the 5259 series.
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models — coverage reported by trajectories, embodiments, skills and institutions.