Short answer
A warehouse pick evaluation should state the task, object set, container, success definition and hold-out axis before any run, then report more than a single success rate: attempts, damage, wrong item, time, intervention and failure causes. Use evaluation items held out by site, SKU family or time, and never mix them back into training.
What this checklist is for
Use it in two situations: when you design an evaluation for your own picking model, and when you judge an evaluation someone else reports or an evaluation set you plan to license. It covers pick-and-place of items into containers. Whole-job metrics, such as orders per hour, depend on the site and are outside its scope. The related task list is in the warehouse task taxonomy.
Published benchmarks give a reference for how such evaluations are built. ARMBench was collected in a warehouse pick-and-place workcell, with images, videos and metadata for more than 235,000 pick-and-place activities on over 190,000 unique objects, and defines tasks for segmentation in clutter, object identification and defect detection (Mitash et al., 2023). Notice that its tasks are perception tasks around picking, not a single success number. A robot-cell benchmark of this kind does not directly evaluate a humanoid's body or whole-task behavior.
Checklist: define before you run
| # | Item | Pass criterion | Evidence to request |
|---|---|---|---|
| 1 | Task definition | Source container, destination, and start/end states written down | Task card with diagrams |
| 2 | Success definition | Item in target, undamaged, correct SKU, within time limit | Written rule; labeled examples of edge cases |
| 3 | Object set | Listed by SKU or category with sizes, weights, materials | Object manifest |
| 4 | Container and clutter | Container type and fill levels specified | Photos at each fill level |
| 5 | Hold-out axis | Held out by site, SKU family, operator or time, matching the claim | Split definition file |
| 6 | Trial count | Chosen by buyer to give useful intervals, with attempts per object stated | Statistical plan |
| 7 | Intervention rule | What counts as a human assist, reset or retry | Written policy; logged interventions |
| 8 | Logging | Per-attempt records with timestamps and outcome codes | Raw logs, video |
| 9 | Independence | Evaluation items never present in training data | Dedup check against training manifest |
A trial count that is "enough" is not universal. It depends on the variance of outcomes and the precision you need, so decide it before running and report intervals.
Checklist: metrics to report
Success rate alone hides what operators care about. Report these together:
- Pick success with the definition from item 2, by object category.
- Attempts per success, including retries.
- Wrong-item and multi-pick rates, which matter because errors in the order stream cost more than a slow pick. See pick errors and mispicks.
- Damage rate, assessed against a written standard.
- Cycle time, split into perception, motion and recovery where possible.
- Intervention rate and cause.
- Failure causes as coded categories: perception miss, grasp slip, collision, planning failure, label unreadable.
The first Amazon Picking Challenge analysis surveyed 26 teams with a 28-question survey and drew lessons that reactive control should be combined with deliberative planning and that the choice of mechanism mainly affected execution speed (Correll et al., 2016). For evaluation, the practical implication is to log failures by cause and by speed. A system that completes picks slowly and one that fails fast need different fixes, and one number cannot separate them.
Checklist: split design
An evaluation set must be held out along the axis your claim requires, and never merged back into training:
- To claim generalization to new items, hold out SKUs or SKU families, not random images of items seen in training.
- To claim generalization to new sites, hold out whole sites, including their lighting, shelving and totes.
- To claim robustness over time, hold out later periods, since packaging and assortments change.
- To claim transfer across operators, hold out people, particularly for human-demonstration data.
Record each item's source site and date so that leakage can be audited. A supplier delivering evaluation material should provide the manifest.
Illustrative evaluation record
# Illustrative only - fields and values are synthetic
trial_id: pick_000417
site_id: site_017
split: eval_heldout_sku_family
task: each_pick_tote_to_tote
object: {sku_family: fam_D, category: polybag, mass_class: light}
container: {type: tote_A, fill_level: medium}
outcome: {result: failure, cause: grasp_slip, wrong_item: false, damaged: false}
timing_s: {perception: 1.1, motion: 4.8, recovery: 0.0}
intervention: none
video: clip_000417.mp4
Red flags and limitations
- A single "success rate" with no definition. Ask how success, damage and retries were judged.
- Random splits. If images of the same SKU appear on both sides, the evaluation measures memory.
- Clean-bin evaluations only. Real totes include overlap, deformable packaging and partial occlusion.
- No failure taxonomy. You cannot improve what you cannot attribute.
- Evaluation inside the supplier's training environment. The evaluation set should come from a different site or period.
- Human-video evaluation presented as robot performance. Human footage can evaluate task understanding or progress estimation; it cannot evaluate a robot's picks.
This protocol does not cover safety validation around people, throughput at scale or integration with warehouse systems. Those require site-specific testing. Evaluation outcomes also apply to the embodiment tested; a result on a robot cell says little about a humanoid.
Worked example: reading a reported result
Suppose a supplier reports that a model achieves a high pick success rate on a warehouse benchmark. Apply the checklist in order. Which task definition applies: tote-to-tote or shelf-to-tote? Which objects: the same SKUs that appeared in training, or a held-out family? How many trials per object, and who decided that a pick had succeeded? Were retries counted as successes?
Suppose the answers are that objects were drawn at random from a pool also used for training, retries counted as successes, and damage was not assessed. The headline rate then measures familiarity with the object pool and tolerance for repeated attempts. It is not evidence of generalization to new SKUs, and it says nothing about product damage. A fair re-run would hold out a SKU family, count a retry as an extra attempt, and report damage under a written standard.
The same discipline applies if you commission an evaluation set. Specify the hold-out axis and the failure taxonomy in the order, and require the supplier to document how each trial was scored.
Training data for the same task is covered separately in each-picking demonstrations; failure categories to score are discussed in warehouse exception handling, and warehouse scan events that can serve as outcome labels in pick confirmation as labels.
Questions to ask suppliers
- How were the evaluation items selected, and how are they separated from training data?
- What is the success definition, and who labeled borderline cases?
- Which hold-out axis is used, and is the split manifest provided?
- Are failures included and coded by cause?
- How many distinct objects, sites and operators appear in the evaluation set?
- Are interventions and retries logged in the records?
- What lighting, shelf and container variation does the set cover?
- Has the same material been used for model development anywhere?
Discuss your requirements
If you need an evaluation set for warehouse picking, describe the tasks, hold-out axes and failure types it must cover to SourceX. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. For wider context see the warehouse data hub, the humanoid training data guide, and the catalog's evaluation sets. The requirements specification template helps structure the request, and troubleshooting evaluation sets shows the same split logic in another domain.
Tell SourceX what your evaluation set needs to cover
Sources
- ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation — warehouse pick-and-place benchmark scale and perception tasks.
- Analysis and Observations from the First Amazon Picking Challenge — survey of competing teams and lessons on control and mechanism.