Skip to content

Privacy and preparation

Physical AI occlusion labels: when the operator's hand hides the action

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

When an operator's hand hides the important moment of a demonstration, label what the camera saw separately from what you infer. Mark the hidden interval as occluded, attach the evidence behind any inferred label, such as before and after states, a second synchronized camera or a machine log, and send clips nothing can settle to review or recapture. Never guess a confident label over hidden frames.

Video and sensors

Physical AI occlusion labels

This page covers physical AI occlusion labels: when the operator's hand hides the action.Illustrative

Key takeaways

  • The key action in a workplace demonstration is often the smallest and the most likely to be hidden by the operator's own hand.
  • Use four visibility states, observed, occluded, out of frame and unknown, and never record occlusion with a tool's display-only hide toggle.
  • Inferred labels are acceptable when graded and backed by time-aligned evidence; unresolved clips stay in the set with their status.
  • A second synchronized camera is the strongest fix, but it films more of the worker and needs consent review.
  • Do not ask operators to change their grip for the camera; fix the cameras and logs instead.

What should you do when a hand hides the step that matters?

Label what the camera saw separately from what you infer happened. Mark the interval where the key action is hidden as occluded, record the evidence that supports any inferred label, such as the before and after state, a second camera or a machine log, and send clips that no evidence can settle to review or recapture. Never let an annotator draw a confident label over frames that show only the back of a hand.

The problem is common in workplace demonstrations because the most important moment is often the smallest. A wire harness connector clicks home inside a closed fist. A clip latches under the thumb. A fastener seats while the palm covers the tool tip. Ego4D's hands and objects benchmark gives this moment a name: annotators mark a pre frame, a point of no return where the object's state change begins, and a post frame, with boxes for hands, tools and objects in each. If the point of no return happens behind the hand, the label that matters most is the one nobody saw.

For a business filming its own operations, the stakes are practical. Footage with silently guessed labels looks complete and fails a buyer's review. Footage with honest occlusion labels and corroborating evidence is smaller on paper and far easier to trust.

Observed, occluded, out of frame or unknown: four states that keep labels honest

Use a small closed set of visibility states for every key object and hand in every interval, and agree on them before annotation starts. Annotation tools already separate these ideas, but they use the words differently, so write your definitions down.

CVAT, a widely used open-source annotation tool, is a good example. Its occluded attribute is for an object hidden by another object or not fully visible; for skeleton points, occluded means the point is still in the frame but behind something. Its outside property marks an object or point that has left the frame, and ends a track when an object disappears. Its hidden property only hides a point for the annotator's convenience and is not saved between sessions, so it must never be used to record occlusion.

Observed, occluded, out of frame or unknown: four states that keep labels honest
StateMeaningWhen to use itTypical tool field
ObservedThe object and the action are visible in this viewThe frames show the contact, motion or state change directlyVisible, no flag
OccludedThe object is in the frame but blocked by a hand, tool, body or other objectThe operator's palm or fingers cover the key contactOccluded attribute
Out of frameThe object or hand has left the camera's field of viewThe action happens below the bench edge or outside a head camera's viewOutside property
UnknownThe view cannot support any state, for example from blur, glare or a dropped frameMotion blur during a fast push, or a lens flareCustom attribute; not the tool's hide toggle

The occlusion annotation and review guide

The guide below is the working procedure for annotators and reviewers. Each step names the annotator's action, the reviewer's check and the field it writes, so a buyer can trace any label back to the decision behind it. Version the guide and record its version on every clip.

The occlusion annotation and review guide
StepAnnotator actionReviewer checkOutput field
1. Name the critical stepMark the interval where the task's state change happens, with pre and post framesInterval matches the task definitioncritical_interval
2. Set visibilityAssign a visibility state to each hand and key object across the intervalStates use the closed list onlyvisibility_track
3. Record the occluderName what blocks the view: own hand, other hand, tool, body, fixture or out of frameOccluder is visible in adjacent framesoccluder_type
4. Label hand contactRecord each hand's contact state where it can be seenNo contact state asserted on occluded frameshand_contact
5. Gather corroborationAttach evidence for the hidden step: pre and post state, second view, machine log, sound, recordEvidence is time-aligned and on fileevidence_refs
6. Grade the inferenceMark the step observed, inferred with strong evidence, inferred with weak evidence or unresolvedGrade follows the evidence rules, not the annotator's confidencelabel_status
7. ReviewA second person checks every inferred or unresolved clip and a sample of observed onesDisagreements go to adjudicationreview_outcome
8. DispositionKeep as observed, keep as inferred, schedule recapture or excludeDisposition fits the buyer briefdisposition

Fields every clip should carry

Each clip should carry enough metadata for a reviewer to reopen the footage and see why a label was given. The list below is the minimum; keep it in a sidecar file next to the video rather than in file names.

At a glance

Fields every clip should carry

  • Clip and task identifiers, the task definition version and the guide version.
  • Camera identifiers, mount position, frame rate and the synchronization offset between cameras.
  • Critical step interval with pre, point-of-no-return and post frame numbers.
  • Visibility state per hand and key object per interval, using the four states.
  • Occluder type for every occluded interval.
  • Hand contact state per hand on visible frames. The hand-object detector from Understanding Human Hands in Contact at Internet Scale uses five: no contact, self contact, other person contact, portable object contact and stationary object contact.
  • Evidence references with timestamps for any inferred label.
  • Label status, reviewer, review date and disposition.
  • Worker visibility: whether faces, badges or screens appear, and the consent or redaction status for each person on camera.

What evidence can stand in for the frames you cannot see?

Evidence can support an inference about a hidden step, but each type proves something different, so record which one you used. Two independent types are better than one.

A second synchronized view is the strongest fix. Ego-Exo4D, for instance, records time-synchronized first-person and third-person video of the same activity, so an action hidden from a head camera can often be seen from a fixed camera, and the reverse. Synchronization is the condition: without a recorded offset, a cross-view label can drift onto the wrong frame.

What evidence can stand in for the frames you cannot see?
EvidenceWhat it can showWhat it cannot showHow to record it
Second synchronized cameraThe hidden contact or motion from another angleAnything also blocked from that angleCamera ID, offset and frame range
Pre and post state framesThat the state changed between two visible momentsHow or exactly when it changedFrame numbers of both states
Tool or machine logThat a tool fired or a station registered completion at a timeWhich hand or grip did itLog source, event and timestamp
Audio cueA click, snap or tool sound near the expected momentThat the sound came from this objectAudio timestamp and description
Work or quality recordThat the unit passed or failed the step laterThe motion that produced the resultRecord ID and result

Illustrative worked example: a connector seated behind the palm

Take an illustrative wire harness station filmed with a head-mounted camera and one fixed overhead camera. The critical step is seating a connector until its latch clicks. In 120 illustrative clips, the head camera shows the latch moment in 73. In the remaining 47, the operator's right palm covers the connector during the point of no return.

For those 47, the overhead camera shows the latch in 29, which become observed in a second view. Of the other 18, 12 have visible pre and post states plus a station continuity test logged within two seconds, so they are labeled inferred with strong evidence. Four have only the post state and are labeled inferred with weak evidence. Two have no usable evidence and are marked unresolved and scheduled for recapture.

A buyer now knows exactly what the set supports. Models learning the seating motion should train on the observed clips. The inferred clips still teach task structure and outcomes, and the occlusion pattern itself, the right palm covering the connector on most clips, is a finding about how the work is actually done.

What variation and recovery behavior is the footage missing?

Occlusion is rarely random; it repeats with the same grip, the same operator and the same step, which means it can hide exactly the variation a buyer wants. Track occlusion rates by step, operator and camera so you can see whether some behaviors are never visible.

At a glance

What variation and recovery behavior is the footage missing

Look in particular for what the hidden frames might contain:

  • Recovery behavior: a re-seat after a failed first push or a regrip, which happens in the same occluded spot as the original action.
  • Operator differences: left-handed operators, or experienced operators who use a different grip, may hide different steps.
  • Failures: a latch that did not click is the same few frames as one that did, so a hidden failure is labeled as success unless evidence says otherwise.
  • Interventions: a supervisor or second worker who reaches in, whose hands may be mistaken for the operator's.
  • Conditions: gloves, lighting changes and part variants that change both visibility and the action.

Why coaching the operator to move their hand is the wrong fix

Do not ask operators to change their grip so the camera can see. The demonstration is valuable because it shows how skilled people actually do the work; a grip altered for the camera may be slower, less safe or simply not the real method, and a model trained on it learns a staged task.

There is one exception. When the buyer's goal is to teach the action itself rather than to record normal operations, a separately labeled set of instructed demonstrations, filmed with a natural and an exposed grip and marked as instructed, can be useful. Keep it apart from normal-operation footage and label it so nobody mixes the two.

The better first fixes are about the cameras: add a fixed view from the side the hand does not cover, raise frame rate for fast steps, and log the station's own completion signal so the hidden moment has a timestamped witness.

Extra cameras, worker visibility and when to stop

Every extra camera that resolves an occlusion also films more of the worker, so camera changes go through the same consent and privacy review as the original collection. A side camera that catches the connector may also catch faces, badges, screens or co-workers who never agreed to be recorded.

Stop or narrow the collection when any of the following is true, and bring in a physical AI researcher and a workplace privacy reviewer before resuming:

  • The critical step is occluded on most clips and no second view or log can be added.
  • A new camera angle would record people who have not consented or cannot be redacted.
  • The station's logs or quality records cannot be time-aligned with the video.
  • Annotators disagree on visibility states more often than they agree after a calibration round.
  • Fixing the view would require changing how the work is done.

Preparing occlusion-labeled footage for licensing

Footage labeled this way is worth more to a buyer than a larger set with guessed labels, because every inferred step can be checked. Package the guide version, the clip fields and the evidence files together, and keep unresolved clips in the delivery with their status rather than dropping them silently.

SourceX can take occlusion-labeled demonstration footage from a manufacturer to an AI developer, running the rights review, contracting, preparation, delivery and payment on the way. A short company assessment, with no video uploaded, shows whether the business qualifies. Large video libraries can stay in the company's own storage or ship on encrypted drives instead of being hosted by SourceX, and nothing is delivered until the company has approved the scope and terms and signed.

Frequently asked questions

Should occluded frames be removed from a physical AI dataset?

Usually not. Keep them with a visibility state and any corroborating evidence, because they show how often a step is hidden and still carry task structure. Remove them only when the brief excludes them or no label can be given at all.

Is occluded the same as out of frame?

No. Occluded means the object is still in the frame but blocked by something such as a hand or tool. Out of frame means it has left the camera's view; CVAT records this with its outside property.

Can annotators infer the hidden action from context?

Yes, if the inference is graded and the evidence is attached. A label inferred from visible before and after states plus a station log is useful; a label inferred from an annotator's expectation alone should be marked unresolved.

Does a second camera solve occlusion?

Often, but not always. A synchronized second view resolves many hidden steps, but it needs a recorded time offset, can be blocked too, and films more of the worker, so it needs consent review.

How should hand contact be labeled when the hand is hidden?

Do not assert a contact state on frames where the hand or object cannot be seen. Record contact on visible frames and let the occluded interval carry its visibility state and evidence instead.

Sources

  • In Ego4D's hands and objects benchmark, annotators label pre, point-of-no-return and post frames for each narrated interaction, with boxes for hands, tools and objects in each frame. Source
  • Ego4D's annotation guidelines define critical frames for each narrated action, including pre-condition, contact, point of no return and post-condition frames. Source
  • CVAT's occluded attribute marks an object occluded by another object or not fully visible in the frame. Source
  • In CVAT, a skeleton point marked outside is out of the frame and saved in annotations, hidden is only an annotator convenience that is not saved, and occluded means the point is still in the frame behind another object. Source
  • In CVAT track mode, an annotator finishes a track with the outside property when the object disappears or becomes too small. Source
  • The hand-object detector from Understanding Human Hands in Contact at Internet Scale labels hand contact as no contact, self contact, other person contact, portable object contact or stationary object contact. Source
  • Ego-Exo4D consists of time-synchronized videos recorded with at least one first-person and one third-person camera. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies