Short answer
A real-world policy evaluation protocol fixes the embodiment, tasks, initial conditions, number of trials, success definition and held-out axes before any policy runs, then reports every trial. Simulated benchmarks such as LIBERO and CALVIN test knowledge transfer and language conditioning, but hardware trials are what show whether a policy works with your robot, sensors and sites.
Policy evaluation is not dataset evaluation
This page is about running a trained policy on hardware and judging it. Judging a dataset before you buy it is a different job, covered in the dataset evaluation checklist. The link between the two is the held-out set: what you hold out of training decides what a trial can claim. See training vs evaluation datasets.
What published benchmarks do and do not give you
- LIBERO is a lifelong-learning benchmark with four task suites totaling 130 tasks, human-teleoperated demonstrations for all tasks and a procedural task-generation pipeline (Liu et al., 2023). It is simulated.
- CALVIN is an open-source simulated benchmark for long-horizon, language-conditioned manipulation that tests novel instructions, environments and objects; its MCIL baseline reaches up to 53.9% on short-horizon tasks (Mees et al., 2021).
- DROID replicated one rig at 13 institutions, covering 564 scenes and 86 tasks, which shows that one setup can be replicated across many sites, as multi-site evaluation of generalization also requires (Khazatsky et al., 2024).
- Open X-Embodiment pooled 22 embodiments and reports positive transfer across robot platforms, so cross-platform claims need cross-platform trials (Open X-Embodiment Collaboration, 2023).
Simulated benchmarks give comparable numbers cheaply. They do not tell you about your gripper, your lighting or your site. Treat them as screening, and put a hardware protocol on top.
Illustrative protocol checklist
| Item | Pass criterion (buyer sets) | Evidence to request |
|---|---|---|
| Embodiment frozen | Robot, end effector, cameras and software version recorded | Config snapshot per session |
| Task definition | Start state, goal state, time limit, allowed retries written down | Task cards |
| Success definition | Binary or staged score, rule for partial credit | Scoring sheet and sample videos |
| Initial conditions | Randomization procedure for object pose and scene | Reset log or seed list |
| Held-out axes | Site, operator, object or time held out as the claim requires | Split manifest |
| Trial count | Fixed in advance, same for each compared policy | Trial table |
| Blind or randomized order | Policies interleaved so drift affects all equally | Run schedule |
| Failure taxonomy | Categories such as grasp miss, timeout, collision, safety stop | Per-trial tags |
| Intervention rules | When a human may reset or intervene, and how it counts | Intervention log |
| Reporting | Every trial, failures included, with video | Trial videos, raw scores |
Illustrative trial record
# Illustrative (synthetic values)
trial_id: t_0412
policy: policy_B
robot_config: cfg_07
site: site_017
operator_reset: operator_A12
scene_seed: 33
task: "move item to bin"
outcome: fail
failure_class: grasp_miss
duration_s: 41
intervened: false
video: vid/t_0412_{cam}.mp4
Design points that change what you can conclude
Hold out along the axis you claim. A claim about new sites needs unseen sites; a claim about new objects needs unseen objects. A policy trained and tested in one lab says little about another.
Fix trial counts before you look. Stopping when a policy looks good inflates results. Pick counts in advance and report variation between sessions.
Control the reset. Different object placements across policies make comparisons unfair. Log the procedure and, where possible, use identical starting layouts.
Separate evaluation data from training data physically. Held-out recordings must not appear in training, tuning or data selection.
Report failures by type. A success rate without failure classes does not tell you whether the next data purchase should target grasping, perception or recovery behavior.
Whether to hold out real data or lean on simulated trials is covered in real vs synthetic and simulation data; how much training data a policy needs is in how many demonstrations published work reports.
What a protocol cannot show
A hardware trial set covers the conditions you ran. It does not certify safety, long-run reliability or behavior at other facilities. Humanoid-specific concerns such as whole-body balance, falls and contact with people need their own safety review outside this page.
Questions to ask an evaluation provider
- Which axes are held out, and how do you prove no leakage into training?
- How many trials per task and per condition, and how was that number chosen?
- Who defines success, and are scorers blind to which policy ran?
- How are resets performed and logged?
- Can you deliver every trial video and per-trial score, not only aggregates?
- How do you handle interventions and timeouts in the score?
- What hardware and software versions were frozen, and for how long?
Sizing and statistics
Success rates from a handful of trials carry wide uncertainty. Decide in advance how much difference between two policies you need to detect, and set trial counts accordingly; report the number of trials next to every rate. Separate per-session variation (lighting, hardware wear, operator) from per-policy variation by interleaving runs. A single lucky day should not decide a purchase.
When comparing against a published result, check that the task definition, reset and success rule match; otherwise the numbers are not comparable even when the task name is the same.
Evaluating a data purchase through policy trials
If the aim is to decide whether a dataset helps, train matched policies with and without it, same architecture and budget, and evaluate both on the same held-out hardware trials. Record the data version, training recipe and seed. This tests the data's usefulness for your robot, which no dataset inspection can do directly. It also shows which failure classes the data reduces, which informs the next request.
Discuss your requirements
If your plan includes held-out recordings for evaluation, say so when you describe your requirements to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. More guidance sits in the procurement hub and the humanoid data guide. See task-understanding evaluation sets for video-based evaluation and cross-embodiment robot datasets for multi-robot data.
Tell SourceX what your evaluation set needs to cover.
Sources
- LIBERO — simulated benchmark structure.
- CALVIN — long-horizon language-conditioned benchmark.
- DROID — multi-institution replicated rig.
- Open X-Embodiment — multi-embodiment pooling and transfer.