Object Tracks and Segmentation Masks in Demonstration Video

The buyer's problem

Hand labels tell a model what the person did. Object labels tell it what moved, where it went and what it looked like at each moment. A policy that must pick up the "blue tote" needs to find it; a progress estimator must know whether the part reached the fixture; a world model benefits from objects that stay the same object across frames.

Annotation vendors can produce masks quickly, but a vague instruction such as "segment all objects" produces labels that vary by annotator and do not match what a model needs. This page gives a specification you can adapt. It covers object-centric labels only; for hand and contact labels see hand-object interaction annotation.

Three label types and who uses them

Published robot-learning work uses the third form. ATM pre-trains a model to predict future trajectories of arbitrary points from video demonstrations and reports use on more than 130 language-conditioned tasks in simulation and the real world (Wen et al., 2023). Track2Act predicts how image points should move toward a goal from web videos of humans and robots, converts the tracks into rigid transforms, and adds a residual closed-loop policy trained from a few embodiment-specific demonstrations (Bharadhwaj et al., 2024).

Mask conventions come from computer vision. COCO collected per-instance segmentation from crowd workers across 91 object categories and about 2.5 million labeled instances (Lin et al., 2014). HOI4D adds panoptic and motion segmentation and category-level object pose to egocentric RGB-D sequences (Liu et al., 2022). For workplace video, motion segmentation, which separates moving from static parts, is often the more useful label.

What these labels can and cannot support

They support detection of task-relevant objects, tracking an item through a workflow, object-state change detection (open or closed, full or empty, placed or not placed), and trajectory-style pretraining. They cannot add 3D pose without depth or fiducials, physical properties such as mass or friction, or robot actions. Masks derived from fixed cameras apply to those viewpoints only; a robot's sensors will see different appearance and occlusion. Object recognition labels are also not manipulation data, a distinction explained in object recognition vs manipulation data.

Illustrative specification

# Illustrative only - values are examples a buyer would set
label_set: object_tracks_masks_v0
sampling:
  masks: every frame at 5 fps keyframes; interpolated masks flagged, reviewed every 10th
  tracks: full frame rate, ID persists across occlusion up to 3 s
  point_tracks: 16 points per manipulated object, queried at first visible frame
taxonomy:
  categories: 40 classes in a supplied list (tote, tray, tool_handheld, fixture, ...)
  stuff_classes: table_surface, floor, shelf (optional)
  unknown_object: allowed, with free-text note
rules:
  hands: separate class "hand"; no merging with held object
  held_object: mask covers visible object only; occluded part excluded
  transparent_and_reflective: label visible silhouette; flag "low_confidence"
  object_split_merge: new ID after separation; parent_id recorded
  out_of_frame: track state = out_of_frame, not deleted
per_instance_fields: track_id, category, mask_rle, visibility_pct, occluded_by, confidence
format: COCO-style JSON for masks; per-video track table (Parquet)
qa: gold_set 2% of clips, double annotation 10%, audit 5% of delivered tracks

Rules that decide label quality

Identity. Define when an object keeps its ID. A box that is opened, emptied and refilled is one container or two? Answer the question in the guideline, with examples, before annotation starts.

Occlusion and contact. In workplace video the hand covers the object during most of the interesting part. Decide whether the mask covers only visible pixels (usual for training segmentation) or an amodal estimate (needed for some state models), and do not mix them.

Sampling. Dense masks on every frame are expensive and often unnecessary. Keyframe masks plus tracker propagation, with flagged propagation and human review, cut cost but need an audit of propagated frames, where drift appears first.

Small and fast objects. Screws, cards and fasteners move quickly at low resolution. If the capture resolution cannot resolve the object, no annotator can fix that; require a resolution check on a sample before labeling.

Taxonomy fit. Use categories that match your downstream tasks. A catalog-level taxonomy of thousands of products is a different job from 40 workplace categories; see annotating existing footage.

Acceptance criteria

1. Mask agreement. On a double-annotated sample, report mask intersection-over-union per category, not one average, with a pass threshold you set.
2. Identity switches. Count ID changes per object per minute on an audited sample.
3. Boundary quality. Review a visual overlay for edge tightness, especially around hands and thin parts.
4. Propagation audit. Check frames far from keyframes.
5. Schema validity. All files parse, IDs are unique, every mask maps to a valid frame.
6. Class balance. Report instances per category and per site so rare classes are visible.

For process-level checks on annotation work, see measuring annotation quality in task video and annotation vendor acceptance criteria.

Handling workplace-specific difficulties

Workplace footage differs from the photo collections that segmentation conventions were built on. Objects are stacked, wrapped, stretched, in motion and partly hidden by hands, gloves and machinery. Decide upfront how to treat stacks (one instance per pallet or per case), liquids and bulk material (stuff-style regions rather than instances), packaging film (visible silhouette plus a flag), and tools that are part of the hand's silhouette during use. Test the guideline on a pilot of a few dozen clips, review disagreements as a group and revise before scaling. Annotator disagreement at pilot stage usually points at an ambiguous rule, not careless work, and fixing the rule is much cheaper than relabeling later.

Questions to ask suppliers

1. Who annotates, and how are they trained on the guideline?
2. Is the process tool-assisted, and how is propagated output reviewed?
3. Can you show a guideline with edge cases and worked examples?
4. How are occluded, reflective and transparent objects handled?
5. What share of frames are human-reviewed versus propagated?
6. How do you measure agreement, and can you share the audit sample?
7. In what format are masks delivered, and are track IDs stable across clips?
8. What do you do when the source video is too low resolution to label?

Annotation work normally leaves rights in the underlying footage unchanged. Labeling does not fix gaps in permission to use the video; see licensing terms for robotics training data. 6D object pose has its own page at 6D pose annotation.

Discuss your requirements

If you need object-centric labels on workplace video, describe the objects, label types and intended use to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. Start from the humanoid training data guide or the workplace task demonstration hub, and see step segmentation annotation for temporal labels.

Discuss your humanoid training-data requirements with SourceX

Sources

• Microsoft COCO: Common Objects in Context — instance segmentation labeling at scale.
• Any-point Trajectory Modeling for Policy Learning — point-trajectory models from video demonstrations.
• HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction — segmentation and pose labels on egocentric sequences.
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation — point tracks from human and robot video.