Depth vs Breadth: Few Tasks in Detail or Many Tasks Lightly

The allocation problem

Every collection budget buys three quantities that compete: the number of distinct tasks, the number of scenes, objects and operators per task, and the number of repetitions of each variant. Adding one usually costs another. A program with a fixed number of recording days has to decide which axis its next day goes to.

This page is about that strategic split. How to cover variation inside one task (objects, start states, failure branches) is the job of the demonstration coverage plan, and the question of volume per task is covered under how many demonstrations. Here the question is upstream: how many different things should the data show at all.

What published datasets chose

Three well-known robot datasets illustrate different choices. They are not comparable on a single scale, because embodiments, control rates and task definitions differ, and the figures below describe what each study collected, not a target.

RT-1 spent its budget on a large catalog of tasks with one platform. DROID kept the task list shorter and pushed variation into scenes and people. Open X-Embodiment pooled existing data to widen embodiments and skills, and reports positive transfer across robot platforms. The lesson for a buyer is that each axis was a deliberate choice by a team with a particular generalization claim, and each study used robot-recorded, teleoperated data that a human-video purchase cannot replace.

The LIBERO benchmark shows the other side. It groups tasks into 4 suites totaling 130 tasks, with human-teleoperated demonstrations for every task, so that researchers can ask which kinds of knowledge transfer between tasks (Liu et al., 2023). Its structure is a reminder that breadth is only useful when the tasks are related in a way the model or evaluation can exploit.

When depth is the right call

Depth fits when:

• the deployment is a short list of tasks at known sites, and failure on any of them blocks release;
• you must show reliability under variation that matters operationally, such as shift, lighting, operator or SKU changes;
• the task has rare branches (jams, misplaced items, reworks) that a thin dataset may not include;
• you need a held-out evaluation inside each task, split by site or operator, which requires enough variety in each task to hold something out.

Depth is also the better first step when the data is the product of a pilot. A narrow pilot exposes problems with camera placement, labeling and consent before they are multiplied across a catalog. See pilot collection scoping.

When breadth is the right call

Breadth fits when:

• the model is a pretrained generalist and the goal is better language grounding, object coverage or task understanding;
• you will fine-tune on your own robot data later and want the pretraining mixture to show a wide range of workplace activity;
• you are using human video for visual representation or latent-action pretraining, where task variety usually matters more than precise repetition;
• the evaluation claim is about unseen tasks, not unseen conditions of known tasks.

Breadth has a cost that is easy to miss. Each additional task needs its own definition of start state, success and failure, and each needs enough examples that annotators apply the same labels. A hundred tasks with inconsistent segmentation can be worse than ten with a clean action taxonomy.

Illustrative allocation worksheet

The worksheet below is a way to write the decision down before scoping. All values are synthetic.

# Illustrative only - values are placeholders, not recommendations
claim: "policy completes tasks T1-T4 at unseen sites"
axes:
  tasks:        {planned: 4,  rationale: "deployment list"}
  sites:        {planned: 6,  rationale: "hold out 2 for evaluation"}
  operators:    {planned: 12, rationale: "hold out 3 for evaluation"}
  object_variants_per_task: {planned: 5}
  failure_branches_per_task: {planned: 2, examples: ["drop", "mis-seat"]}
budget_split:
  train_depth_pool: "tasks T1-T4, all sites except 2 held out"
  breadth_pool:     "adjacent tasks T5-T20, lighter labeling, pretraining only"
  evaluation_pool:  "held out by site and operator, never mixed back"

Two features matter more than the numbers. The breadth pool is labeled differently from the depth pool, so cost per hour differs. The evaluation pool is defined before collection, so the split is not decided after the fact.

What this strategy choice cannot settle

No amount of task breadth turns video into robot control data. If your data is human workplace video, the breadth you buy supports visual and task understanding, and the control signal still has to come from robot, teleoperated or instrumented data, as explained in what human demonstration video can and cannot do.

Published scale figures also do not transfer to your program. They depend on architecture, embodiment, control rate and the claim being tested, and the studies above say little about your tasks. Breadth also does not fix a coverage gap inside a task: ten kitchens with the same happy path still have no failure recoveries. Finally, breadth across sites multiplies permission work, since each site's consent, notice and licensing position has to be understood separately; see multi-site collection standardization.

Decision table

Questions to ask suppliers

1. How many distinct tasks does the material cover, and who defined the task boundaries?
2. How many sites, operators and object variants appear within each task, not only in total?
3. Is the distribution flat across tasks, or do a few tasks dominate the hours?
4. Do all tasks use the same camera setup and labeling scheme?
5. Can we hold out whole sites or operators for evaluation, and are identifiers provided to do so?
6. Which tasks include failures, retries or corrections, and how are they labeled?
7. Can a smaller pilot be drawn from the same source to test labeling before a larger purchase?
8. How does cost or licensing scope change between a narrow, deep package and a wide, shallow one?
9. Are permissions the same across sites, or does each site or contributor have its own terms?

Discuss your requirements

If you are deciding how to split a budget between a short list of deployment tasks and a wider pretraining pool, describe the tasks, modalities and permitted uses to SourceX. SourceX looks for US businesses that hold the material you describe, checks the data and the supplier's licensing permissions, and coordinates license and delivery. A request does not guarantee a match; SourceX follows up about potential sources. For the wider map, start at the humanoid training data guide or the workplace task demonstration hub, and see the requirements specification template. The site's page on robotics and embodied AI data gives the broader context.

Discuss your humanoid training-data requirements with SourceX

Sources

• Open X-Embodiment: Robotic Learning Datasets and RT-X Models — pooled multi-embodiment dataset and cross-platform transfer.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — scenes, tasks and collectors in one replicated rig.
• RT-1: Robotics Transformer for Real-World Control at Scale — task-catalog breadth with a single robot design.
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning — task suites for studying transfer.