Short answer
A 6D pose label gives an object's 3D rotation and translation in a camera or world frame. It needs a 3D model of the object, calibrated cameras and a rule for symmetric objects, so pose data cannot be created from catalog photos alone. Specify the model source, calibration files, symmetry definitions, an error metric and an independent accuracy check.
Why pose labels are a separate purchase
Detection says an object is in the image and where in the picture it sits. Pose says how it is oriented and how far away it is, in a frame a planner can use: the handle points up and 14 cm to the left of the gripper. Many workplace tasks, such as placing a part in a fixture or lifting a tool by its grip, depend on that. Pose labels are also the most expensive label type in the objects cluster, because each one ties an image to a physical model and a camera geometry.
This page is a specification for buyers who need those labels from an existing archive, from a new capture session, or from a mix. For the difference between recognizing and handling objects, read object recognition data vs manipulation training data first.
What published benchmarks establish
Two benchmark papers show what a defensible pose dataset contains. The BOP benchmark brings eight datasets into a unified format, including two new datasets with varying lighting, and defines a pose-error function that handles ambiguous poses; the authors evaluated 15 methods, and point-pair-feature methods performed best in that comparison (Hodan et al., 2018). T-LESS offers 30 industry-relevant texture-less objects, many of them symmetric or similar to one another and some parts of others, recorded with three synchronized sensors (structured-light RGB-D, time-of-flight RGB-D and a high-resolution RGB camera), with roughly 39,000 training and 10,000 test images per sensor and a CAD model and a reconstructed model for each object (Hodan et al., 2017).
Both results point the same way: pose data without models, calibrated sensors and a metric that accounts for symmetry is hard to use. T-LESS also shows why industrial parts are difficult: with no texture, a model must rely on shape, and many parts differ only in small details.
What pose data can and cannot support
It can support pose estimators for rigid objects, grasp planning that starts from a known model and a detected pose, placement and insertion checks, and evaluation of estimators on held-out scenes.
It cannot cover objects without a rigid model (cables, bags, apparel; see deformable product properties), and it says nothing about weight, friction or how an item behaves when moved. A pose label says where the object was, not where to grip it; that needs grasp annotations or robot trials. Training on rendered images from a CAD model can supplement real data, but the transfer to real cameras should be tested rather than assumed; see synthetic renders vs real images.
Prerequisites a supplier must meet
| Prerequisite | Why it is needed | What to request |
|---|---|---|
| 3D model per object | The pose is defined relative to a model's coordinate frame | Mesh or CAD, units, origin definition, license |
| Camera intrinsics | Converts pixels to rays | Focal length, principal point, distortion per camera |
| Extrinsics or world frame | Places objects relative to the robot or a table | Transform files and calibration date |
| Depth alignment | RGB-D labels need depth registered to color | Registration method and error |
| Symmetry definitions | Many parts have equivalent poses | Symmetry group per object |
| Annotation method | Determines label accuracy | Manual fitting, fiducial rig, motion capture or model-based refinement |
Where the model comes from matters. Manufacturers may supply CAD, which is exact but may include internal geometry and carry engineering-file rights; scanned models match real wear but include scan error. See manufacturer 3D models and CAD and point clouds and depth scans.
Illustrative specification
This block is Illustrative; names and thresholds are examples.
# Illustrative 6D pose label spec - synthetic values
object_set: 40 rigid parts from a tool crib (obj_001 ... obj_040)
models:
format: textured mesh, millimeter units
origin: documented per object; axes drawn in a reference sheet
symmetry: {obj_007: continuous_about_z, obj_012: discrete_2_fold, obj_020: none}
sensors:
- {id: cam_A, type: RGB-D, resolution: 1280x720, intrinsics: calib/cam_A.json}
- {id: cam_B, type: RGB, resolution: 4096x3000, intrinsics: calib/cam_B.json}
labels_per_instance:
pose: {R: 3x3, t: 3-vector (mm), frame: camera}
visibility_fraction: 0-1
mask_visible: required
occluded_by: object_id list
scene_variation: [lighting x 3, clutter level x 3, background x 2, tilted views]
accuracy_check:
method: independent fiducial or refit on 5% sample
report: median and 95th-percentile error, per object
metric: symmetry-aware pose error, as in published benchmarks
splits: train / validation / test, split by scene arrangement
Acceptance tests
- Reprojection check. Render each model at its labeled pose over the image; sample 200 instances and flag any visible misfit.
- Independent accuracy audit. A subset labeled by a second method or fiducials, with reported translation and rotation error per object.
- Symmetry audit. Confirm that symmetric objects are scored with their equivalent poses and not penalized for flips.
- Visibility stratification. Results should be reportable by occlusion level; instances nearly hidden should be marked, not silently dropped or kept as ground truth.
- Calibration validity. Calibration dates must fall inside the capture dates, and cameras that moved must have new files.
- Split integrity. Scenes in the test set must not share arrangements with training; this affects the evaluation described in object recognition evaluation metrics.
Choosing the annotation method
Four methods are in common use, and each trades cost against accuracy and flexibility. Manual model fitting, where an annotator aligns a rendered model with the image, works on existing images but is slow and varies by person. Marker or fiducial rigs give accurate poses quickly but leave markers in view unless they are hidden or removed in a second pass. Motion-capture or robot-arm-referenced setups give poses in a known frame but need an instrumented space. Model-based refinement from a rough start reduces effort but inherits the refiner's errors, so it still needs the independent audit described above.
If masks are also required, see segmentation masks for products. Ask which method produced which share of the labels, because a mix is normal and the accuracy differs by method. Also ask whether any scenes were captured with the object moved between views while the camera stayed fixed, since that pattern can propagate a single carefully fitted pose to many frames.
Limitations and extra preparation
Pose accuracy is bounded by the model's accuracy and the calibration's accuracy; label quality cannot exceed either. Labels in a camera frame must be transformed to the robot's frame using hand-eye calibration, which comes from the robot setup, not from the data supplier. Existing business imagery rarely has calibration, so it can be used for detection but seldom for pose without new capture. Annotating images by hand-fitting a model is slow and varies by annotator; instrumented capture with fixtures or markers is faster but may show the markers. Workplace items may be proprietary designs, so permissions for models and images need to be agreed separately.
Questions to ask suppliers
- Which objects have a 3D model, who made it, and under what license?
- How were poses obtained: manual fitting, markers, motion capture or automatic refinement?
- Can you provide intrinsics and extrinsics for every capture session, with dates?
- How are symmetric and near-symmetric parts defined in the labels?
- What accuracy did you measure against an independent reference?
- Do images include clutter, partial occlusion and varied lighting, or only isolated parts on a turntable?
- Can the pose labels be delivered separately from raw images, if images carry restrictions?
- How are objects that changed design between sessions handled?
Discuss your requirements
If you already know the object list and the models you hold, describe the pose requirement to SourceX. SourceX reviews the requirement and follows up about potential sources, such as US businesses that hold part models and imagery; annotation and specialist capture partners can be arranged case by case, and a request does not guarantee a match. See also the humanoid training data guide and the object data hub.
Discuss your humanoid training-data requirements with SourceX
Sources
- BOP: Benchmark for 6D Object Pose Estimation — unified pose datasets, symmetry-aware error and method comparison.
- T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects — industrial texture-less objects, multi-sensor capture and per-object models.