Long-Horizon Multi-Step Task Demonstrations

The buyer's problem

A single skill, such as grasping a cup, is short and has a clear end. Real work chains skills: gather supplies, move, open, place, check, close, reset. Models that handle such chains need data in which the chain is intact, and buyers often discover that their clips are trimmed to single steps, with the connections cut out.

Long-horizon performance is tested in benchmarks built for it. CALVIN is an open-source simulated benchmark for long-horizon, language-conditioned manipulation that tests on novel instructions, environments and objects; its MCIL baseline reaches up to 53.9% on short-horizon tasks (Mees et al., 2021). The benchmark exists to measure how a policy copes when tasks are strung together under new instructions and scenes, and the figure shows how much room its baseline left.

Task anatomy of a long-horizon demonstration

A useful way to specify a long task is as a hierarchy.

Ego4D Goal-Step shows the hierarchical idea at scale, with goal annotations for 2,807 hours of video and 48K procedural step segments covering 430 hours (Song et al., 2023). Ego-Exo4D records skilled activities with first-person and several third-person cameras at the same time, which helps annotators resolve what is happening when one view is blocked (Grauman et al., 2023).

For robot data, a mobile bimanual system trained with whole-body teleoperation used 50 demonstrations per task, and co-training with static datasets raised success by up to 90% (Fu et al., 2024). Those figures describe one platform and its tasks. They are a useful reference for how a single long task may be demonstrated repeatedly, not a target for yours.

What to capture

• Whole runs. Record from the moment the worker starts the goal until it is finished or abandoned, with no gaps. Resets between runs should be recorded or at least logged.
• Consistent clock and ID. Every clip carries run, step and operator IDs so you can reassemble sequences.
• Multiple viewpoints. A head-mounted camera for hands and objects, plus a fixed view that places the worker in the room. See multi-view synchronized task recording.
• Variation by design. Change layout, object positions, starting state and order of independent steps across runs.
• Recovery coverage. Planned and natural recoveries: a dropped item, a missing supply, a repeated step. Failure and recovery demonstrations covers how to source them.

What the data cannot support

Human long-horizon video teaches ordering, preconditions and what completion looks like. It does not give a humanoid the joint-level actions to carry out each step, and an error early in a chain can carry through later steps in ways a labeled video does not reveal. Robot or teleoperated data for the primitive skills remains necessary, with video supplying the structure above them. Treat the video as supporting a planner or high-level policy; for that division, see workplace video for task planning.

Coverage is the second limit. A dozen perfect runs of one task at one site tell you little about recovery or variety. Use a demonstration coverage plan to list the conditions, objects and failure modes you need, then count runs against it.

Choosing the hierarchy depth

More levels are not always better. Each added level multiplies annotation cost and disagreement. Pick the depth from the model that will use it.

A planner that sequences skills needs goals and steps whose names match the skills it can call. Atomic actions add little. A video-language model may benefit from step text and goal text, but not from atomic action boundaries. A policy trained from robot demonstrations needs the step boundaries mainly to cut and condition training segments, and recovery links to learn what to do after an error. Decide which of these you are buying for before you commission labels, and write the taxonomy first, with a short list of the steps that must be distinguishable.

Order flexibility needs explicit treatment. Some tasks have a strict sequence, and some allow steps to be swapped or done in parallel. FineBio, a dataset of mock biological experiments, annotates protocols, steps and atomic operations and notes strict step order alongside flexible order of atomic operations (Yagi et al., 2024). Record which steps are order-constrained in your schema, so that a legitimate reordering is not scored as an error.

Illustrative long-horizon record

# Illustrative only - synthetic values
run_id: run_site_009_0114
goal: "restock supply closet, floor 2"
duration_min: 24.6
hierarchy:
  stages:
    - {name: gather, start: 0.0, end: 5.3}
    - {name: transport, start: 5.3, end: 9.8}
    - {name: shelve, start: 9.8, end: 22.4}
    - {name: close_out, start: 22.4, end: 24.6}
  steps_total: 31
  events:
    - {t: 14.2, type: "item_dropped", recovered: true, recovery_step: "pick up, re-place"}
language: {goal_text: yes, step_text: yes, narration: no}
outcome: completed_with_recovery
camera: {head: 30fps, fixed: 30fps}
sync: shared timecode, drift check at start and end

Quality and acceptance criteria

Examples a buyer can set: no unexplained gaps longer than a set number of seconds within a run; step boundaries labeled by two annotators with agreement reported against a gold set; at least some fraction of runs containing a labeled recovery event; every stage appears across more than one operator and one site; start and end states recorded. Check hierarchy consistency by sampling: do step segments sum to the stage, and do stages sum to the run? See step segmentation annotation for method.

Suppliers and sourcing routes

Operators who run the same multi-step routine daily, such as housekeeping, restocking, lab prep or kitchen setup, are the natural suppliers. Existing footage seldom preserves full runs. A commissioned session could be scripted to include variations and recoveries, with a collection provider handling consent, rigs and quality review, if such a collection proves feasible. Compare routes in existing footage vs commissioned recording.

Questions to ask suppliers

1. Are recordings whole runs, or excerpts? How do you mark cuts?
2. How do you record resets and the start state?
3. Can you vary layout and start condition on purpose?
4. How are interruptions, errors and recoveries recorded and labeled?
5. What identifiers link clips to the same run?
6. How many operators and sites will appear for each task?
7. Who defines the step taxonomy and who resolves disagreements?
8. How long can a person work before quality drops, and how do you manage it?

Discuss your requirements

If you need multi-step tasks recorded end to end, describe the task chain, variation and label levels to SourceX. SourceX reviews the requirement and follows up about potential sources; commissioned collection is scoped case by case and a request does not guarantee a match. Wider context is in the demonstration hub and the humanoid training data guide.

Discuss your humanoid training-data requirements with SourceX

Sources

• CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — CALVIN long-horizon language-conditioned benchmark.
• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — Ego4D Goal-Step hierarchical goal and step annotations.
• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives — Ego-Exo4D multi-view skilled-activity recording with expert commentary.
• Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation — Mobile ALOHA whole-body teleoperation, 50 demonstrations per task.
• FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation — FineBio hierarchical annotation and step-order structure.