Short answer
A workplace benchmark object set is a documented collection of physical items, bought in identical copies, that lets different robots and different test dates face the same objects. Choose items that span the shapes, sizes, weights and materials of your tasks, record their properties, scan them, and plan replacements. The set supports comparison, but it does not define the tasks or the scoring.
What a benchmark object set is for
Evaluating a manipulation system on whatever objects happen to be in the lab makes results impossible to reproduce. A fixed set fixes one source of variation, so a change in score reflects the policy and not the mug. It also allows a second team, or the same team a year later, to repeat a test, and it lets a buyer compare vendors on the same items.
The reference point is the Yale-CMU-Berkeley (YCB) object set, proposed for benchmarking robotic manipulation, prosthetics and rehabilitation research. It consists of everyday objects that vary in shape, size, texture, weight and rigidity, and is accompanied by a database of high-resolution RGB-D scans, physical properties and geometric models, along with example benchmarking task protocols (Calli et al., 2015). A workplace set borrows the idea, not the contents: your items come from your tasks.
This page is a checklist for building and accepting such a set. The question of testing on objects never seen in training is covered in novel object generalization evaluation; here the focus is on the physical collection.
What a benchmark set can and cannot support
It can support repeatable task trials, comparisons between systems, controlled tests of how performance changes with object property (heavier, smaller, slicker), and matched simulation experiments if models are made.
It cannot define a benchmark by itself. Objects without a task protocol, start-state rules, success criteria and trial counts give no comparable numbers. A set also covers a sample of your world: it cannot represent every item, the variation between production lots, or the clutter and wear of a working site. And performance on the set is evidence about those objects; claims about novel objects require held-out items, kept out of any training or tuning. Keep training and evaluation roles separate, because an object set used for both loses its value as evaluation.
Checklist: selecting objects
For each candidate, pass or fail against these criteria. Thresholds are set by the buyer.
| Criterion | Pass condition | Evidence to record |
|---|---|---|
| Task relevance | Item appears in, or stands for, a class in your target tasks | Task list reference |
| Property spread | Set spans shapes, sizes, masses, rigidity and surface finish | Property table with ranges |
| Gripper range | Includes items near the limits of your hands or grippers | Size and mass vs gripper spec |
| Difficulty tiers | Easy, medium and hard items are labeled by reason | Tier definition |
| Availability | Identical copies can be reordered for years | Part number, supplier, lead time |
| Stability | Item does not change with temperature, humidity or handling | Material note |
| Safety | No item poses unmanaged risk to people or equipment | Hazard note; see hazard properties of objects |
| Redistribution | Item can be shared with collaborators and vendors | Rights note |
Decide which items come from your long-tail needs (long-tail object coverage) and which are common reference items. Include some items that stress perception: transparent, reflective, texture-less or dark objects (transparent and reflective objects). Include a few that stress manipulation but not perception: heavy, slippery, deformable. Keep these tiers separate so a failure can be attributed.
Checklist: documenting the set
- Identifier and source. Manufacturer, part number, supplier, order date, lot or batch.
- Measured properties. Mass, dimensions, center of mass where relevant, surface friction estimate, rigidity class. Measured values, not catalog nominal values; see physical property annotations.
- Images and scans. Multi-view RGB, depth scans and a mesh with a stated coordinate frame and units. Google Scanned Objects shows what a scanned household-item collection looks like: more than one thousand items, a Creative Commons license and preprocessing for the Ignition Gazebo and Bullet simulators (Downs et al., 2022).
- Model source. Scanned, reconstructed or from manufacturer CAD (manufacturer 3D models and CAD).
- Simulation readiness. Collision geometry, mass and friction parameters if the set is also used in simulation (simulation-ready object assets).
- Handling instructions. Storage, cleaning, replacement rules and a retirement threshold.
- Datasheet. A written account of purpose, composition and recommended uses, following Gebru et al..
Illustrative set specification
The spec below is Illustrative; item classes and sizes are synthetic.
# Illustrative workplace benchmark object set - synthetic values
set_id: wbos_v0
purpose: pick-and-place and tool-handover evaluation in a parts crib
copies_per_item: 3 (1 in use, 1 reference scan, 1 spare)
items:
- {id: o_001, class: rigid_box, tier: easy, mass_g: measured, finish: matte}
- {id: o_014, class: cylindrical_bottle, tier: medium, finish: glossy_transparent}
- {id: o_022, class: hand_tool, tier: hard, reason: asymmetric_mass}
- {id: o_031, class: bagged_part_kit, tier: hard, reason: deformable}
heldout_for_novelty: [o_040 ... o_048] # never used in training or tuning
per_item_files: [mesh.obj, rgbd_scans/, properties.json, photos/, license.txt]
protocol_refs: [task_protocols_v1.md]
replacement_policy: replace on visible wear, label damage, or mass change over 2%
Acceptance checks on delivery
- Count and identity. Each delivered item matches the order, part number and lot.
- Measured vs declared. Spot-measure mass and size on a sample; record deviations.
- Scan quality. Meshes render correctly, are watertight where needed, and align with the physical item at a documented scale.
- License check. The terms for models, images and any CAD allow your intended uses, including sharing the set with vendors.
- Spare strategy. Duplicates exist so that damage does not end a study.
- Held-out integrity. Items reserved for novelty tests are isolated from development.
Sourcing
Because the suppliers here are manufacturers and distributors, procurement is mostly buying physical goods, and rights risk is low for the objects themselves. Manufacturers may also supply CAD or product models for items they make, and distributors can supply identical copies with consistent part numbers. Models and imagery of those items carry their own rights, covered in product image rights. Public sets and scans, such as YCB and Google Scanned Objects, offer a reference but may not match your tasks. Decide between buying existing assets and commissioning scans or imaging using choosing a sourcing route.
Red flags
- Items that cannot be reordered in the same form.
- A set chosen to make a particular robot look good.
- Catalog weights treated as measured weights.
- No held-out items.
- No task protocol attached to the set.
- Models with no stated units or coordinate frame.
- Objects that are hazardous with no handling plan.
Limitations and extra preparation
Wear changes objects over time. Production lots vary. A set designed for one site's tasks may not represent another's. Scoring requires its own protocol, including trial counts and resets; see object recognition evaluation metrics for recognition-side measures. Plan for storage space and a custodian.
Questions to ask suppliers
- Can you guarantee identical copies of each item for the next several years, and what is the notice period for a design change?
- Do you hold CAD or scan data for the items, and under what license?
- What lot or batch information can you provide for each shipment?
- Can you supply mass and dimensions measured on shipped units?
- Are there restrictions on shipping, storage or sharing for any item?
- Which items have variants that look alike but differ in weight or finish?
- Can you supply a discontinued-item warning list for the set?
- Who holds rights to product photographs you provide?
Discuss your requirements
Start with the list of tasks and the object classes they involve. When you describe the benchmark object set to SourceX, SourceX reviews the requirement and follows up about potential sources, which may include US manufacturers and distributors that hold models or imagery. A request does not guarantee a match, and nothing is contracted until a supplier agrees. For wider context, see the humanoid training data guide and the object data hub.
Tell SourceX what your evaluation set needs to cover
Sources
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols — the YCB object set, models, properties and protocols.
- Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items — scanned household items, license and simulator preprocessing.
- Datasheets for Datasets — documenting composition and recommended uses.