Short answer
A training dataset teaches a model; an evaluation dataset measures it, and it only measures anything if the model never trained on it. Buy them as separate deliverables with their own specification, license terms and access controls, and hold evaluation data out along the axis your claim needs: site, operator, object, time or embodiment. Recorded held-out episodes are not the same thing as robot rollouts.
Two purchases, not one split
Many programs buy one corpus and carve a test set out of it. That is cheap, and it answers a narrow question: how does the model do on more of the same? A deployment claim is different. "Works at a new site", "handles packaging it has not seen" and "still works on the night shift" are claims about conditions outside the training data. A random slice of the corpus cannot test them, because neighboring clips share the site, the operator, the lighting and often the objects.
Treat evaluation data as its own purchase. It usually needs stricter labels, a tighter permissions record and a handling regime that keeps it out of every training job. The requirements specification template has a holdout block; this page explains how to fill it.
| Dimension | Training dataset | Evaluation dataset |
|---|---|---|
| Job | Fit model parameters | Estimate behavior under conditions you name |
| Selection | Breadth: many conditions, including noisy cases | Designed around the claim; each condition appears on purpose |
| Label rigor | Some noise is tolerable if the method and volume absorb it | Label errors distort the result directly; double annotation and adjudication are common choices |
| Access | Training jobs and data engineers | Evaluation harness only, read-only, named users, logged |
| Reuse | Used in many runs | Every round of tuning against it spends some of its value |
| Rights | Training and fine-tuning | Evaluation-only use, confidentiality over items, publication terms |
| Cost of an error | A weaker model | A wrong decision to deploy, release or fund |
Hold out along the axis of your claim
The unit of separation is whatever your claim says will be new. Decide it before anything is collected.
| If your claim is | Hold out by | What leaks if you split by clip instead |
|---|---|---|
| Works at sites it never saw | Whole site, including layout and lighting | Same shelving, floor and camera mounts on both sides |
| Works for new workers or operators | Operator or demonstrator | Individual speed, handedness and motion style |
| Handles unseen items | Object instance, category or product family, as you define "new" | The same instances in different poses |
| Holds up over time | A later time window or a post-change period | Same packaging runs and seasonal conditions |
| Works on a different robot | Embodiment | Same kinematics, controller and camera rig |
| Follows new instructions | Task composition or instruction wording | Paraphrases of training instructions |
Sequential data makes random frame-level or clip-level splits especially weak: adjacent segments of one recording are near-duplicates. Declare the holdout unit to the supplier up front, so those sites, operators or objects are collected, stored and delivered as separate lots. Splitting a delivery afterward is hard because the factor you wanted to hold out is usually mixed through every file.
How leakage happens in practice
- Shared sessions. Adjacent segments, re-encoded copies or augmented versions of one recording land on both sides of the split.
- Shared identity. The same site, operator or object instance appears in training and test, so the score measures familiarity rather than skill.
- Pretraining overlap. A model initialized from a video or robot pretraining corpus may have seen your evaluation sites or items, especially if the corpus came from the same suppliers. This is benchmark contamination.
- Derived artifacts. Pose estimates, captions, embeddings or synthetic variants made from evaluation items and stored with training artifacts carry the items into training.
- Tuning to the test. Choosing hyperparameters or checkpoints by repeated scores on one set turns it into a validation set. Keep three roles apart: training, validation for tuning, and a final evaluation set that is touched rarely.
- Supplier reuse. If a supplier sells or publishes the same items elsewhere, others may train on them, and a later training purchase from that supplier may contain them.
The glossary defines held-out data in general terms. The checks to run when a delivery arrives, including a cross-check that no holdout ID appears in a training lot, are in acceptance testing for dataset deliveries.
Recorded evaluation sets and robot rollouts
Embodied work has two kinds of evaluation, and they are different purchases.
Recorded evaluation sets are held-out episodes or clips scored offline: step recognition, progress estimation, language grounding, action prediction error. They are repeatable and can be bought as a dataset. They do not measure closed-loop task success on a robot, because a deployed policy visits states that recorded demonstrations never contained.
Rollout evaluations run the policy, in simulation or on hardware, from defined initial conditions under a success criterion. What you buy or build is a protocol: tasks, scene setup, object set, reset procedure, number of trials and scoring. That is covered in real-world policy evaluation protocols.
Published benchmarks follow the same logic. LIBERO packages 4 task suites totaling 130 tasks, with human-teleoperated demonstrations for all of them and a procedural pipeline for generating tasks (Liu et al., 2023). CALVIN is a simulated benchmark for long-horizon, language-conditioned manipulation that tests on novel instructions, environments and objects (Mees et al., 2021). Each defines evaluation through tasks and conditions as well as data files. Public benchmarks give comparability with other work; they do not reproduce your sites, objects or robot. If a benchmark's data is public, also check that none of it sits in your pretraining mix.
Illustrative evaluation-set specification
# Illustrative only — synthetic IDs; every value is an example the buyer sets
eval_set: EVAL_2026_003
version: v1.0 # frozen; any change creates v1.1
claim_tested: generalization to unseen sites and unseen operators, task family tote_unpacking
holdout_unit: [site, operator]
holdout_values:
sites: [site_017, site_021]
operators: [operator_A12, operator_A19, operator_B03]
excluded_from_training:
- every recording, frame, pose estimate, caption and embedding derived from holdout values
- any recording from holdout sites made on other dates or by other suppliers
content: recorded_episodes # or: rollout_protocol with scene setup, reset steps, success criteria
labels: double-annotated; disagreements adjudicated; guideline v2.1
documentation: data card (intended use, collection process, split design, known gaps)
access: read-only evaluation harness; named users; access log
use_rights: internal evaluation only; no training, fine-tuning or distillation; no redistribution
publication: aggregate results may be published; items may not be quoted without consent
refresh: retire or replace after a buyer-set number of evaluation rounds or on suspected exposure
reporting: per-site and per-operator results, not one average
Documentation that keeps an evaluation honest
Ask for a data card for the evaluation set. Data Cards are structured summaries of the essential facts about a dataset that stakeholders need across its lifecycle, and their authors report lessons from deploying over 20 of them (Pushkarna et al., 2022). For an evaluation set the useful fields are intended use, collection process, how the split was designed and what the set does not cover. Fields for robotics data are in datasheets for robotics datasets.
Borrow from the other side too. Model cards are short documents that accompany a trained model and report performance across groups, intended use contexts and evaluation procedures (Mitchell et al., 2018). Applied to robot policies, that means reporting results per site, operator or object stratum and recording which version of the evaluation set produced each number.
Licensing and handling an evaluation set
An evaluation set reveals the conditions you consider hard, so a leak both contaminates the test and exposes your plan. Keep its license separate from the training license and ask for:
- evaluation-only use, with an explicit bar on training, fine-tuning and distillation, and a statement of how that bar applies to derived data and model weights;
- confidentiality over the items, and control over whether results that quote items may be published;
- a limit on the supplier selling or publishing the same items to others, or at least notice if it does; see exclusivity and field of use;
- the same consent and third-party review as any other lot, because evaluation footage shows people too.
The broader clause list is in licensing terms for robotics training data.
What this guidance cannot settle
- Size. How many items, trials and conditions an evaluation needs depends on your architecture, task, embodiment and the size of the difference you must detect. No universal minimum exists; use a pilot and report uncertainty.
- Third-party contamination. You control your own pipeline and your supplier terms. You cannot see what is in someone else's pretraining corpus.
- Aging. Process changes, new packaging and new robot revisions erode an evaluation set; budget for refreshes.
- Coverage by design. A held-out set only tests what it contains. For planning which conditions to include, see the demonstration coverage plan, and for unseen items the novel-object generalization evaluation checklist.
Questions to ask suppliers
- Which identifiers (site, operator, device, object, date) are recorded for every item, so I can hold out by the unit my claim needs?
- Can the evaluation lot be collected, stored and delivered separately from training lots, ideally from different sites or operators?
- Have any items in this lot been sold, shared or published elsewhere, and will you tell me before that happens?
- How were evaluation labels produced and checked, and can they be double-annotated with adjudication?
- Do you keep copies of the items, or pose estimates, captions and embeddings derived from them, and where?
- If you also supply training data, how do you keep its sources apart from this lot's?
- What documentation covers intended use, collection process and known gaps for this lot?
- Will the license limit use to evaluation and give me confidentiality over items and a say over publication?
Tell SourceX what your evaluation set needs to cover
The SourceX buyers page leads to a request form that includes evaluation criteria. Describe the evaluation set as a requirement of its own, with its holdout unit, coverage and handling rules. SourceX looks for US businesses that hold the data you describe, checks the data and the supplier's licensing permissions, agrees allowed uses in a license and coordinates delivery. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and nothing is contracted until a supplier agrees. For the wider picture, see the procurement hub and the humanoid training data guide. The catalog page on evaluation sets covers the general case.
Tell SourceX what your evaluation set needs to cover
Sources
- Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset summaries and lessons from deploying over 20 Data Cards.
- Model Cards for Model Reporting — reporting performance by group, intended use and evaluation procedure.
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning — 4 task suites, 130 tasks, teleoperated demonstrations, procedural generation.
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — simulated benchmark testing novel instructions, environments and objects.