Short answer
A novel-object evaluation set tests a robot on items it never saw in training. Define what 'novel' means for your claim (new instance, new category, new material or new packaging), hold those items out from every training source, check public and pretraining data for leaks, keep the physical items or exact records obtainable, and report results by difficulty stratum. The checklist below gives pass criteria and the evidence to request.
Why novelty has to be defined before items are chosen
"Generalizes to new objects" is a common claim in robot learning and an easy one to inflate. A cereal box from a brand the model has seen, in a size it has not, is novel in one sense and familiar in every way that matters. A "new category" may already sit in the vocabulary the vision backbone was pretrained on. Results only mean something when the type of novelty is stated and the holdout enforces it.
Published work shows why the split must be reported. Dex-Net 2.0 reports 99% precision on 40 novel objects, a separate figure from its 93% success on 8 known adversarial objects (Mahler et al., 2017); the two numbers answer different questions. The general rules for keeping evaluation data separate are on training vs evaluation datasets; this page applies them to objects.
Types of novelty
| Novelty type | Example | Holdout rule | Typical leak |
|---|---|---|---|
| New instance, seen category | An unseen brand of dish soap | Hold out by SKU and brand family | The item's packshot sits in a catalog gallery used for training |
| New category | Power tools, never in training | Hold out by taxonomy node | Category present in backbone pretraining data |
| New material or appearance | Transparent or black items | Hold out by material attribute | Similar items in synthetic data |
| New packaging version | A redesign of a seen SKU | Hold out by artwork version and date | Old artwork nearly identical |
| New geometry for grasping | Unseen shapes and sizes | Hold out by shape class | Same mesh in a public 3D library |
| New context | Seen items at a new site | Hold out by site | This is domain shift, not object novelty; report it separately |
Which level your claim needs depends on whether the robot must recognize exact SKUs or categories; instance vs category recognition covers that choice.
The checklist
A. Claim and definition
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Novelty type stated | One or more types from the table, written down before item selection | Evaluation plan |
| Comparison set | A matched set of seen items, scored under the same protocol | Item lists for both sets |
| Task stated | Recognition, pose, grasping or full manipulation, each scored separately | Protocol document |
B. Holdout integrity
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Item exclusion | No held-out item appears in any training image, mesh, demonstration or gallery | Exclusion list matched against training manifests |
| Sibling exclusion | Variant groups held out together where the claim requires it | Variant-group table |
| Supplier separation | Where possible, held-out items come from a source not used for training | Source per item |
C. Contamination checks
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Public 3D libraries | Held-out items checked against libraries used in simulation | Search log against each library |
| Pretraining vocabularies | "New category" claims checked against backbone pretraining categories | Category comparison |
| Web catalogs | Item images searched in any scraped or licensed web data | Search method and results |
Public 3D libraries are large. Objaverse holds more than 800,000 3D models sourced from Sketchfab (Deitke et al., 2022), and Google Scanned Objects provides over one thousand scanned household items prepared for simulation (Downs et al., 2022); if your simulation or synthetic data drew on either, check your held-out items against them. Category vocabularies overlap too: OmniObject3D's 190 categories overlap ImageNet and LVIS (Wu et al., 2023), a reminder that everyday categories are rarely new to a pretrained backbone. See also benchmark contamination.
D. Reproducibility
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Physical availability | Items can be bought again, or enough units are held for future runs | Purchase records; stock held |
| Exact identification | GTIN, lot and packaging version recorded per unit | Item register with photos |
| Models and scans | A mesh or scan per item for pose evaluation | Files and capture notes |
| Stable protocol | Placement, lighting and clutter are scripted | Protocol with layout diagrams |
YCB set the pattern of distributing a physical object set together with models and example benchmarking protocols (Calli et al., 2015). For workplace items, the equivalent is covered in workplace benchmark object set. SKU churn works against reproducibility: a retail item can vanish within a year, so buy spare units.
E. Coverage and difficulty
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Strata | Items grouped by difficulty (size, material, clutter, look-alikes) | Stratum assignment rules |
| Counts per stratum | Enough items per stratum to report separately, at a level you set | Counts table |
| Long tail | Rare and awkward items included deliberately | Selection rationale |
MetaGraspNet separates real annotated images into five difficulty levels and adds an unseen-object set (Gilles et al., 2022), a structure worth copying. Selecting awkward items on purpose is covered in long-tail object coverage.
F. Scoring and governance
| Check | Pass criterion | Evidence to request |
|---|---|---|
| Metrics fixed in advance | Metric, threshold and aggregation chosen before testing | Protocol version |
| Symmetric objects handled | Pose errors account for symmetry | Metric definition |
| Sealed access | Held-out images and item lists kept from training teams | Access log |
| Versioning and retirement | Set versioned; items retired once exposed | Change log |
For pose, reuse conventions rather than inventing them: BOP defines a unified dataset format, a pose-error function that handles ambiguous poses and an online evaluation system (Hodan et al., 2018). Metric choice for recognition is covered on object recognition evaluation metrics, and physical trial design on real-world policy evaluation protocols.
Illustrative evaluation set manifest
# Illustrative only — synthetic items, counts and identifiers
eval_set: novel_objects_retail_v1
claim: "new-instance and new-material generalization for shelf picking"
novelty_types: [new_instance_seen_category, new_material]
seen_comparison_set: 60 items, same categories, same protocol
held_out_items: 90
strata:
standard_rigid: 40
transparent_or_reflective: 20
deformable_bags: 15
look_alike_variants: 15
per_item: {gtin, lot, packaging_version, purchase_date, units_held: 3, scan: mesh_obj, photos: 6}
sources: launches after 2026-06 from retailer_R22 (not a training supplier)
contamination_checks: [public_3d_libraries, training_galleries, scraped_web_index]
access: images and item list held by eval team only; training team sees aggregate scores
retirement: items exposed in any published figure move to training after next release
Red flags
- "Novel" items chosen after training, by people who saw the training data.
- Held-out items that are variants of training items, presented as new categories.
- A single aggregate score with no breakdown by stratum.
- No record of which lots or packaging versions were tested.
- Evaluation images stored where training pipelines can read them.
Limitations
Novelty is relative to everything a model was trained on, including foundation-model pretraining you do not control, so some claims cannot be fully verified. Small held-out sets give wide uncertainty; report counts with results. A recognition holdout does not test manipulation: an item can be recognized perfectly and still be impossible to grasp, and grasp or manipulation novelty needs trials on the robot.
Questions to ask suppliers
- Can you supply items launched after a given date, with launch dates documented?
- Can you provide several physical units per item from the same lot?
- Which of these items appear in catalogs, image sets or 3D assets you have licensed to others?
- Can you record GTIN, lot and packaging version for each unit shipped?
- Do you carry transparent, reflective, deformable or look-alike items in the categories we name?
- Will you keep supplying the same items for repeat evaluations, or tell us when they are discontinued?
- Can items be delivered without catalog images, so they stay unseen by our training pipeline?
Discuss your requirements
Tell SourceX what your evaluation set needs to cover: the novelty types, categories, strata and number of units per item. Send the evaluation requirement to SourceX; it looks for US businesses that hold suitable items or records, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX reviews each request and follows up about potential sources; a request does not guarantee a match. See also the product and object data hub, the humanoid training data guide, held-out data and the catalog's private evaluation sets.
Tell SourceX what your evaluation set needs to cover
Sources
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols — physical object set with models and protocols.
- BOP: Benchmark for 6D Object Pose Estimation — unified format and symmetry-aware pose error.
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics — separate results on novel and known objects.
- MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis — difficulty levels and an unseen-object set.
- OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation — categories overlapping common vocabularies.
- Objaverse: A Universe of Annotated 3D Objects — large public 3D pool to check for leaks.
- Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items — scanned items commonly used in simulation.