Workplace Video for High-Level Task Planning Models

How planning models use video differently

A planner sits above the controller. Given a goal such as "reset the room" and an image, it proposes the next step or a sequence of steps, which a lower-level skill then executes. In SayCan, a language model supplies task knowledge while pretrained robot skills constrain its proposals to what is feasible in the scene, and the authors report that this grounding roughly halves errors against ungrounded baselines in a real kitchen (Ichter et al., 2022). ProgPrompt takes the other route, listing the available actions and objects as a program-like specification plus example programs so the model generates plans over a known vocabulary (Singh et al., 2022).

Both results point at the same data need. A planner has to learn what a task consists of, in what order, with what preconditions, and what the scene looks like when a step is done. Video of people working shows exactly that. It does not show how a gripper closes on a handle. That distinction is the reason this page is separate from low-level imitation data, discussed in demonstrations for imitation vs pretraining.

Embodied reasoning models extend the same idea to perception. Gemini Robotics-ER is described as handling pointing, grasp and trajectory prediction, multi-view correspondence and 3D boxes, and sits beside a vision-language-action model that controls robots directly (Google DeepMind, 2025). Reasoning of this kind benefits from video that has grounded labels on objects and scene state, not only from captions.

What planning data looks like

Hierarchical labels are the established way to describe this. Ego4D Goal-Step added goal annotations for 2,807 hours of Ego4D video and 48K procedural step segments covering 430 hours, built on a data-driven taxonomy (Song et al., 2023). Robot demonstrations of the same long tasks, with recovery coverage, are a separate purchase described in long-horizon task demonstrations.

What this data cannot support

• Control. Planning labels tell a model that "open the cabinet" follows "pick up the tray." They give no joint targets, grasp strategy or force.
• Feasibility for your robot. A person can complete a step that your humanoid cannot reach, lift or grip. SayCan's design exists because language alone does not know what the robot can do. Your own skill library and affordance model must filter the plan.
• Safety judgment. A plan that was fine for a trained worker may be unsafe for a robot near people.
• Evaluation of generalization. Training on one franchise's way of doing a task will not show whether a planner handles another site's layout. Hold out by site; see task-understanding evaluation sets.

A procedure document can supply the intended order and constraints, but not the physical reality. What SOPs can and cannot teach a humanoid covers that boundary, and pairing video with documents is covered in instruction-paired demonstration datasets.

Sourcing routes

Existing footage. Training sessions, process-monitoring cameras and quality reviews are the likeliest archives. Planning use tolerates a fixed camera better than manipulation use does, since the goal is step recognition rather than hand trajectories. The cost is uneven length: archives often hold edited clips that skip transitions, which removes exactly the context a planner needs. Ask for unedited recordings where they exist, and see continuous shift recordings vs curated clips for the trade.

Annotation of what exists. If footage is suitable but unlabeled, a goal-and-step pass is realistic; see adding annotations to existing footage for the limits.

Commissioned recording. If you need controlled variants, such as the same task with different layouts or deliberate errors, record new sessions under a task script. Scope the pilot using pilot collection scoping.

Evaluating a sample before you buy

Planning data is easy to misjudge from a highlight reel. Ask the supplier for ten whole recordings chosen by you, and check four things.

First, can you reconstruct the task from the video without reading the label? If an annotator needed the SOP to know what happened, the visual evidence is thin, and a model trained on it will lean on language alone. Second, do step end-states show up in the frame? A planner learns to verify completion from what the scene looks like, so steps whose result happens off-screen give weak supervision. Third, compare the written procedure against the footage and count deviations; a procedure that is followed in every clip suggests staged recording or heavy editing. Fourth, check that the vocabulary of objects and actions is stable. If one site says "turn over" and another says "reset" for the same task, you will need a normalization pass before training.

Also decide how the labels will be used. A language-model planner that is prompted with an action list, as in ProgPrompt, needs a closed set of verbs matching your robot's skills. A model that learns from video-language pairs can use freer text. Fixing the action vocabulary first, using an action taxonomy for workplace video, avoids relabeling later.

Illustrative planning-oriented package

# Illustrative only - synthetic fields and values
clip_id: site_017_clip_0042
goal: "turn over guest room 214"
duration_min: 31.5
steps:
  - {id: s1, label: "strip bed linen", start: 0.0, end: 4.2, end_state: "bed bare"}
  - {id: s2, label: "bag soiled linen", start: 4.2, end_state: "bag at door", end: 5.1}
  - {id: s3, label: "wipe bathroom surfaces", start: 5.1, end: 14.8, end_state: "surfaces dry"}
deviations: [{type: "repeat", step: s3, reason: "visible residue"}]
language:
  instruction: "Reset the room for the next guest."
  step_text: per step, annotator-written
procedure_ref: SOP-HK-07 (supplier document, licensed separately)
camera: ceiling_fixed_1080p_30fps
excluded: joint states, actions, forces

Questions to ask suppliers

1. Is each recording a whole task, or an edited excerpt? Are cuts marked?
2. Can clips be tied to job, work-order or procedure identifiers?
3. Does the written procedure match what workers actually did? Where do they diverge?
4. How many sites, layouts and operators appear for the same task?
5. Are rework, interruptions and recoveries retained?
6. Who wrote the step labels, using what taxonomy?
7. Can I license the procedure text alongside the video, and is it the supplier's own?
8. What do notices to recorded workers say about use in AI training?

Discuss your requirements

If your planner needs whole-task video with goal and step labels, describe the tasks, label structure and permitted uses to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and licensing permissions, and follows up about potential sources; a request does not guarantee a match. Wider context is in the workplace task demonstration hub and the humanoid training data guide.

Describe the workplace demonstration data your robotics project needs

Sources

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances — SayCan grounds language-model plans in feasible robot skills.
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models — ProgPrompt specifies available actions and objects for plan generation.
• Gemini Robotics: Bringing AI into the Physical World — Gemini Robotics-ER embodied reasoning capabilities.
• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — Ego4D Goal-Step hierarchical goal and step annotations.