Short answer
Eye-gaze is an optional channel in egocentric task demonstrations: it records where the demonstrator looked, which can help with attention modeling, intent and action recognition. It needs eye-tracking hardware, per-person calibration and extra consent, and it adds no robot actions or forces. Buy it only if a specific model or analysis will use it.
The buyer's problem
Head-mounted video shows what was in front of the demonstrator, but not what they were attending to. When a worker reaches for a part, the camera frame holds the whole bench; gaze marks the item that mattered. That extra signal is attractive, and it is also easy to over-buy: it raises hardware, calibration and privacy costs, and many pipelines never read it.
This page helps you decide whether a gaze channel belongs in your head-mounted capture specification, and what it must include if it does.
What published datasets record
Several research datasets pair first-person video with gaze, and they show the channel in practice.
EGTEA Gaze+ provides first-person videos with gaze tracking data, hand masks and action annotations. Its authors model gaze as a probabilistic variable that guides attention for action recognition (Li et al., 2020). That is a recognition use: gaze helps decide what action is happening.
Ego-Exo4D records skilled activities with one egocentric and several exocentric cameras simultaneously, along with eye gaze, IMU, multichannel audio, 3D point clouds and camera poses (Grauman et al., 2023). Gaze there is one channel among many, which suits research that studies multiple signals together.
The Project Aria glasses used in several such efforts include two eye-tracking cameras with infrared illumination, alongside a rolling-shutter RGB camera, SLAM cameras, two IMUs and a microphone array (Engel et al., 2023). If a collector uses research glasses, ask which sensors are enabled for your recording, since some channels may be off by default or restricted.
What gaze can support
| Use | How gaze helps | Caveat |
|---|---|---|
| Action and intent recognition | Narrows the object of attention before a hand moves | Needs aligned labels to measure the effect |
| Attention-guided perception | Supplies a spatial prior on where to look | The robot's gaze policy differs from a human's |
| Step segmentation | Fixations on tools or parts often precede steps | Not every step has a visible fixation |
| Annotation support | Directs annotators to the relevant object | Adds a dependency on gaze quality |
| Proficiency analysis | Compares expert and novice looking patterns | Needs enough demonstrators per skill level |
| Active-perception research | Studies when and where people look to resolve uncertainty | Humanoid camera motion differs from human eye movement |
Gaze is a human physiological signal. A robot has no eyes that saccade in the same way, so anything you learn from gaze must be translated into a robot-side mechanism, such as a region-of-interest prior or a head-motion target. Treat it as an auxiliary signal, not a drop-in for robot actions.
What gaze cannot support
- Robot actions. Gaze is not an action space and gives no joint commands.
- Forces and contact. It says nothing about grip, pressure or slip.
- Reliable attention in all cases. People look at things they are not manipulating, and fixation is not intention.
- Robot camera design. It does not specify the right sensor layout; see matching camera viewpoint to humanoid sensors.
Collection implications
Gaze is rarely available in existing business footage. It almost always means a new recording with eye-tracking hardware, so it belongs in the commissioned route; see existing footage vs commissioned recording.
- Hardware. An eye-tracking headset or glasses, with the scene camera and gaze camera synchronized.
- Calibration. Each wearer needs calibration at the start, and again after the device shifts. Drift over a shift can degrade accuracy.
- Environment. Strong sunlight, reflective glasses and prescription lenses can degrade tracking.
- Wearer variation. Facial shape, eyelashes and glasses change tracking quality; log it per person.
- Time sync. Gaze samples run at a different rate than video frames; deliver raw samples with timestamps rather than only per-frame coordinates.
- Cost. Extra devices, longer setup per demonstrator and more QA per session.
Pair it with a recruiting plan for demonstrators that includes time for calibration, and consider adding IMU data from the same device for head-motion context.
Permissions
Gaze is behavioral data about a person's attention. It can reveal what a worker reads, notices or avoids, so it deserves its own line in consent: what is collected, who sees it, how long it is kept and whether it can be used for anything beyond model training. Face-adjacent eye-camera images may also be biometric-adjacent in some jurisdictions; ask counsel. This is not legal advice. For the wider consent picture, see worker consent and identifiable people.
Illustrative gaze channel specification
# Illustrative only - synthetic values, set by the buyer
gaze:
device: "wearable eye tracker with scene camera"
raw_samples: {rate: "buyer sets", fields: [t_device, x_norm, y_norm, confidence]}
calibration: {per_session: true, check_after_min: 30, accept_if_error_below: "buyer sets"}
time_sync: "shared clock with scene camera; offset logged per session"
delivered: [raw_samples, per_frame_projection, calibration_report]
exclusions: [eye_images] # retain only if consented and needed
qa: "reject segments with confidence below buyer threshold"
metadata: [wearer_id (pseudonymous), eyewear_type, lighting_note, site_id]
Gaze, viewpoint and labels
Gaze comes from the wearer, so it belongs to the egocentric view. If your recordings also include fixed cameras, the gaze point must be projected into those views through calibration, which adds error; compare the trade-offs in egocentric vs exocentric workplace video.
On the labeling side, gaze can reduce effort by pointing annotators to the object of interest, but it can also bias them toward what the demonstrator looked at rather than what they touched. For contact labels, hand-object interaction annotation remains the primary source. Run a small pilot with and without gaze overlays and measure whether annotation speed or agreement changes before paying for the channel at volume.
Questions to ask suppliers
- Which eye-tracking hardware will be used, and at what sample rate and scene-camera resolution?
- How is calibration performed and checked, and how often within a session?
- What share of samples have usable confidence, and how do you report it per demonstrator?
- Are raw gaze samples delivered, or only gaze projected into video frames?
- How are gaze and video clocks synchronized, and what offset is logged?
- Will eye images be stored or deleted after gaze is computed?
- Does consent from the demonstrators cover gaze data specifically, including retention and reuse?
- How do glasses, lighting or workplace PPE affect tracking, and how will failed segments be flagged?
- Can you provide a pilot sample so my team can test the gaze channel against our own pipeline before a full order?
Describe your needs
If your plan needs gaze alongside egocentric task video, describe the tasks, devices and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; it can scope a commissioned collection of new recordings if no existing asset fits, and a request does not guarantee a match. For wider navigation, see the workplace task demonstration hub and the humanoid training data guide. The catalog entry on first-person video of manual work gives related context.
Describe the workplace demonstration data your robotics project needs
Sources
- Project Aria: A New Tool for Egocentric Multi-Modal AI Research — Aria sensor set including eye-tracking cameras.
- In the Eye of the Beholder: Gaze and Actions in First Person Video — EGTEA Gaze+ and gaze as an attention variable for action recognition.
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — multichannel recordings including eye gaze and IMU.