Short answer
Judge object recognition data for robots by whether errors would change what the robot does, not by one aggregate accuracy score. Ask for per-class results on rare items, confusions between look-alike products, pose error that respects symmetry, performance under occlusion and lighting, and a held-out split by object, site and time. Aggregate scores can look strong while the failures that matter stay hidden.
Why one accuracy number misleads a robot program
A recognition model with a high average score can still be wrong every time on the one item your workflow cannot afford to confuse. Averages weight common classes heavily, so a long tail of rare items barely moves the score. A robot also acts on its answer: a wrong label on a bottle of cleaning fluid sends a different object to a different place than a wrong label on a sponge.
So the evaluation question is not "how accurate is this dataset's baseline?" It is "which errors would this data let through, and what would each one cost downstream?" The checklist below turns that into evidence you can request from a supplier before you license anything.
This page covers evaluation. For the earlier decision of whether recognition data is the right purchase at all, see object recognition data vs manipulation training data.
What the standard benchmarks actually measure
Public benchmarks show what a metric was designed to answer. The COCO benchmark covers 328k images with about 2.5 million labeled instances across 91 object categories, with per-instance segmentation collected from crowd workers (Lin et al., 2014). Those categories are broad ("bottle", "cup"), which suits general detection research. A warehouse, kitchen or maintenance bay often needs instance-level or SKU-level identity, which a category benchmark does not score.
The BOP benchmark takes the opposite emphasis: 6D pose of rigid objects from a single RGB-D image, with eight datasets in a unified format, an evaluation of 15 methods, and a pose-error function designed to handle ambiguous poses (Hodan et al., 2018). The lesson for buyers is that a pose metric must account for symmetric or near-symmetric objects, or it will penalize correct answers and reward lucky ones.
Neither benchmark tells you how a model will behave on your objects, in your lighting, from your robot's cameras. Treat them as templates for how to report, not as evidence for your use.
Errors that matter downstream
| Recognition error | Why a robot cares | What to measure |
|---|---|---|
| Look-alike confusion (same shape, different variant) | Wrong item picked or wrong handling applied | Confusion matrix restricted to look-alike groups |
| Missed detection of a small or partly hidden item | Item ignored, collision, or incomplete task | Recall by object size and occlusion level |
| Right class, wrong pose | Grasp planned on the wrong face or axis | Pose error with symmetry handling, not just detection |
| Loose or shifted mask edges | Grasp points land on the background | Boundary quality near graspable regions |
| Over-confident wrong answers | No chance to stop or ask for help | Calibration: confidence versus actual correctness |
| Right object, wrong state (open, empty, damaged) | Task proceeds on a false assumption | State-level labels and their own error rates |
Object state matters enough to deserve its own treatment; see object state recognition.
Illustrative acceptance checklist
The thresholds are examples a buyer sets for their own task; the page does not suggest any number is adequate in general.
| # | Check | Pass looks like (example) | Evidence to request |
|---|---|---|---|
| 1 | Class coverage | Every object in your workflow list appears with enough varied views for you to test it | Per-class image counts and view counts |
| 2 | Per-class results | Recall and precision reported for each class, rare ones included | Table by class, not only a mean |
| 3 | Look-alike groups | Confusions inside groups you define stay below your tolerance | Confusion matrix for named groups |
| 4 | Instance or category | Labels match the identity level your robot needs | Label schema and an identity-level sample |
| 5 | Pose, if needed | Error reported with symmetry handled; ambiguity flagged | Pose metric definition and per-object results |
| 6 | Conditions | Results split by lighting, occlusion, clutter and distance | Stratified results plus capture metadata |
| 7 | Held-out design | Test objects, sites and time periods absent from training | Split manifest with the unit of separation |
| 8 | Label quality | Sampled labels audited by someone other than the annotator | Audit sample, error rate, disagreement notes |
| 9 | Your cameras | A sample captured, or re-checked, on sensors close to your robot's | Camera model, resolution, mount, height |
| 10 | Documentation | Collection, composition and limits written down | A datasheet or data card |
Documentation frameworks such as Datasheets for Datasets and Data Cards give suppliers a structure for item 10, though neither is a quality guarantee.
Red flags
- One headline accuracy figure with no split, class table or confidence interval.
- The evaluation images come from the same photo sessions, shelves or lots as the training images.
- "Novel object" claims where the same product appears under a different SKU or pack size on both sides of the split. Held-out design is covered in novel object generalization evaluation.
- Pose results that ignore symmetry or report only detection hits.
- Labels produced by a model and never audited by a person.
- No account of how items were chosen, so a convenient subset may be passing as a representative one.
Limitations: what recognition metrics cannot tell you
Even a rigorous recognition evaluation answers only whether the perception layer sees the right thing. It does not show how an object behaves when handled: how it slips, deforms, or resists. Grasp success, placement accuracy and recovery from a drop need robot-side evidence, which comes from robot or teleoperated interaction rather than from images.
Metrics also age. Packaging redesigns and catalog changes shift appearance, so a score measured last quarter describes a catalog that may have moved. Plan a refresh for items that matter; see long-tail object coverage for ways to handle rare items.
Finally, a good score on a supplier's test set does not transfer to your cameras and sites. Keep a small private set from your own environment, and keep it out of training. If you need 6D labels as part of the offer, check what the annotation involves in 6D pose annotation.
Choosing identity level and building your own check set
Decide early whether the robot needs category recognition ("a bottle"), instance recognition ("this SKU") or both; the choice changes every metric above, and instance vs category recognition sets out the trade-offs. Then keep a small private set drawn from your own workplace, such as the workplace benchmark object set, captured on your cameras and never shared with a supplier. Re-score the supplier's data against it before and after any model change, so you can tell whether the data or the model moved the number.
Questions to ask suppliers
- How were the evaluation images separated from training images: by object, site, session or time?
- Can you provide per-class and per-condition results, including the rarest classes?
- Which look-alike groups exist in the catalog, and how are confusions within them scored?
- Who produced the labels, who audited them, and what was the sampled error rate?
- If poses are labeled, how are symmetric objects handled in the metric?
- Which cameras, resolutions and heights were used, and how do they relate to a humanoid's head or wrist sensors?
- Does any image appear more than once, including crops, rotations or re-exports?
- Are the same physical items present on both sides of any split?
- Which items have been redesigned or discontinued since capture?
- What documentation exists on collection method, composition and known gaps?
Discuss your requirements
If you can describe the objects, conditions and error types your evaluation has to cover, tell SourceX what your evaluation set needs to cover. SourceX reviews the requirement and follows up about potential US business sources; a request does not guarantee a match. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case.
For the wider map, see the humanoid training data guide and the product and object data hub. The catalog entry for images and inspection photos and the evaluation sets page describe related buying paths.
Tell SourceX what your evaluation set needs to cover
Sources
- Microsoft COCO: Common Objects in Context — scale and category granularity of a general detection benchmark.
- BOP: Benchmark for 6D Object Pose Estimation — 6D pose benchmark with ambiguity-aware pose error.
- Datasheets for Datasets — structure for documenting dataset composition and collection.
- Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset summaries.