Short answer
Request a sample that is drawn the way your full purchase would be, not a curated highlight reel, and review it against your own specification before discussing price. A useful sample includes raw files, the schema, metadata, label examples and a datasheet. Check what it reveals about coverage, synchronization, permissions and failure cases, and what it cannot tell you.
The buyer's problem
Samples are among the cheapest diligence a data buyer gets, and among the easiest to waste. A supplier's chosen sample is likely to show the supplier's best case. A sample that is too small hides variation, and one delivered in a different format from the full dataset hides conversion problems. This page is a checklist for requesting existing-dataset samples and reviewing what arrives.
It covers samples of data that already exists. Pilot collections, where new recordings are made to a buyer's script, are a different exercise covered under demonstrations; start with existing workplace footage vs commissioned demonstration recording. For the full evaluation of an offer, use the dataset evaluation checklist. Sample review feeds that checklist; it does not replace acceptance testing for deliveries.
Step 1: Define what the sample must answer
Before writing to a supplier, write down the questions the sample should settle. Typical ones:
- Does the file format load in our pipeline without custom work?
- Are all streams present and synchronized (video, depth, proprioception, force, audio, labels)?
- Is the metadata sufficient to split by site, operator, object and time?
- Do labels match the label definition in the specification?
- How often do failures, retries and edge cases occur?
- Are permissions documented at the same granularity as the data?
If a question cannot be answered from a sample, say so and plan another check, such as a metadata export or a supplier call. A requirements specification makes these questions concrete.
Step 2: Specify how the sample is drawn
Ask for the sampling method in writing. Choices from weakest to strongest evidence:
| Sampling method | What it shows | Weakness |
|---|---|---|
| Supplier-selected highlights | Best-case quality | Hides variation and defects |
| Random draw by buyer from an index | Typical quality | Needs a manifest first |
| Stratified draw (by site, operator, object, lighting) | Coverage across strata | Requires accurate metadata |
| Draw that includes the worst-labeled and shortest episodes | Failure handling | Supplier must allow it |
Request a manifest first: a list of file or episode identifiers with metadata but no content. Select the sample from it, so the supplier does not choose. If a manifest cannot be shared before agreement, ask for a random draw made by a method you can inspect.
Step 3: Illustrative sample request
Illustrative sample request - placeholder values, adapt to your specification
1. Manifest (no content): episode_id, site_id, operator_id (pseudonymous),
date, duration, sensors present, label status, permission category.
2. Draw: 30 episodes chosen by the buyer from the manifest, stratified by site_id,
plus 5 episodes with label status "disputed" or "corrected".
3. Format: the same container, naming and compression as the full delivery.
4. Include: schema file, calibration files, timestamps, data dictionary,
label guidelines, annotator instructions, datasheet or data card.
5. Permissions: for each sampled episode, the permission category and its record.
6. Handling: sample provided under a limited evaluation license; deletion on request
after the review period; no model training on the sample unless agreed in writing.
The last line matters. Samples are data too. Decide in advance whether evaluation includes training on it, and put it in the sample terms; see licensing terms for robotics training data.
Step 4: Review checklist
Mark each item pass, fail or not-determinable, and keep the evidence.
Format and loading
- Sample loads with the buyer's reader; any required conversion is documented. See robot learning data formats.
- Schema matches the data dictionary; field types and units are stated.
Streams and time
- Every promised modality is present in the sampled episodes.
- Timestamps are monotonic, with a stated clock source; see sensor synchronization requirements.
- Calibration files are present and plausible; see calibration requirements.
Content
- Viewpoints show the task-relevant hands, tools and objects.
- Labels match their definitions on a manual spot check, with disagreements counted.
- Failures, retries and recoveries appear, not just clean runs.
- Variation across sites, operators and objects matches the manifest's claim.
Rights and provenance
- Each sampled item has a documented source and permission category; see provenance and chain of custody.
- No visible third-party identifiers, documents or screens appear that the license does not cover; see confidential business information in footage.
What a sample cannot tell you
- Dataset size or total diversity. Thirty episodes cannot show the tail of a large archive.
- Training value. Whether data improves a model is established by training experiments, which depend on architecture and task.
- Consistency over time. Pipelines change; early episodes may differ from late ones unless the draw spans the period.
- Rights beyond the sampled items. A clean permission record on thirty items says little about the rest without an audit of the rest.
- Held-out suitability. Whether a set can serve for evaluation depends on how it is split; see training vs evaluation datasets.
Documentation to request alongside
Datasheets for Datasets proposed that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses (Gebru et al., 2018). A supplier unable to answer those questions for a sample has probably not documented the full dataset either. See datasheets and data cards for robotics datasets for the physical-data adaptation.
Red flags
- The supplier will only send files they selected, and cannot explain how.
- Sample format differs from the format in the offer.
- Missing streams, with the explanation that they exist "in the full set".
- No answer about permissions for specific items.
- Pressure to decide before the review is complete.
After the review
Write a short decision record for each sample: what was requested, how it was drawn, which checklist items passed or failed, and what remains undetermined. Share the failures with the supplier and watch the response. A supplier that fixes a labeling inconsistency and re-delivers a corrected sample with a changelog is showing the behavior you will need during a full delivery. One that disputes every finding, or replaces the sample with a cleaner one, is showing you the opposite. Keep the sample and the record under the retention rules in the sample terms, and use the same checklist against any later delivery so the comparison is like for like.
Questions to ask suppliers
- Can you provide a manifest with site, operator, date, sensors and label status before a sample is chosen?
- May we select the sampled episodes ourselves?
- Will the sample be in the same format, compression and naming as delivery?
- Which streams exist for every episode, and which only for some?
- What is the label error rate you measured, and how?
- Can the sample include episodes with failures or corrected labels?
- What permission records can be shown for the sampled items?
- What are the terms for the sample itself: use, retention, deletion?
- What documentation, such as a datasheet or data card, accompanies the full dataset?
Discuss your requirements
If you are comparing candidate sources, describe the dataset kind, modalities and evaluation criteria to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match; SourceX follows up about potential sources. See the humanoid training data guide and the procurement hub.
Tell SourceX what your evaluation set needs to cover
Sources
- Datasheets for Datasets — Datasheets for Datasets: documentation of motivation, composition, collection process and recommended uses.