Short answer
Public instruction datasets such as ALFRED and BEHAVIOR-1K center on household and everyday activities, so they are valuable format and benchmark references but do not cover workplace vocabulary, procedures, tools, roles or constraints. A buyer targeting workplace deployment should treat them as design references and specify separate workplace instruction data, paired with robot or teleoperation data for control.
The decision this page helps with
Teams building instruction-following robots often start from public benchmarks, because they are free, documented and comparable across papers. When the target is a plant, warehouse, kitchen, clinic or repair shop, the question becomes how far those benchmarks carry. This page does not argue that household data is useless. It explains what each public source covers, where the gap sits, and what to specify when workplace instruction data is the missing piece.
What the two public references cover
ALFRED maps natural-language instructions and egocentric vision to action sequences for household tasks. It pairs expert demonstrations in interactive visual scenes with about 25,000 natural-language directives, written both as high-level goals and as low-level step instructions (Shridhar et al., 2019).
BEHAVIOR-1K offers 1,000 everyday activities across 50 scenes in a simulator that models rigid objects, deformable objects and liquids. Its activity list was drawn from candidates sourced from time-use surveys and WikiHow, and a survey of 1,461 respondents rated preferences to rank them (Li et al., 2024).
Neither covers a workplace. They show how to pair instructions with demonstrations, how to split goals from steps, and how to pick a representative activity list. Those are reusable ideas. The content of the instructions, however, belongs to the home. For this page, both are only format and labeling references.
Where the gap shows up
| Dimension | Household benchmark pattern | What a workplace deployment tends to need |
|---|---|---|
| Vocabulary | Common nouns and verbs a layperson uses | Part numbers, equipment names, shorthand, abbreviations |
| Instruction authors | Crowd annotators writing from video | Supervisors, trainers, procedure authors with their own conventions |
| Structure | Short goals and step lists | Preconditions, branches, exceptions, sign-offs |
| Constraints | Implicit common sense | Explicit rules about sequence, safety, quality and authorization |
| Objects | Everyday items, mostly one of each | Many near-identical SKUs, tools, fixtures and consumables |
| Success | Task goal reached in a scene | Specification met, record completed, handoff accepted |
| Environment | A home layout | Site-specific layouts, signage, restricted areas |
This table is a reasoned comparison, not a measurement. A buyer should test whether the gap matters for their tasks, for example by evaluating a model on a small held-out set of real workplace instructions. Instruction-following evaluation for embodied tasks describes how to design such tests.
What workplace instruction data can look like
Workplace instructions come in several forms, and they are not interchangeable. Procedure documents give formal step order; verbal directions from a supervisor are looser and context-dependent; labels, signs and checklists add short constraints. A model that follows a polished SOP may fail on how a person actually says "grab the next tote and stage it." See natural worker instructions for that distinction, and procedural text for instruction grounding for how formal text is used.
Pairing text with demonstrations is the strongest pattern, since it ties words to behavior; see instruction-paired demonstration datasets. For defining the tasks themselves, public occupational resources can help with scoping. O*NET, for instance, includes task statements and detailed work activities, with a file mapping tasks to activities (O*NET Resource Center). See also occupational task taxonomies. It is a vocabulary for scoping, not a source of instructions, and it says nothing about any specific site's procedure.
Workplace terms often differ from site to site. A workplace terminology glossary from the supplier helps a model, or an annotator, map local words to standard ones.
Illustrative specification for a workplace instruction set
This is an Illustrative specification with synthetic values. A buyer would adapt it to their architecture and tasks.
# Illustrative only - not an available dataset
workplace_instruction_set_v0:
domain: kitchen_prep (example)
instruction_types: [formal_procedure_step, spoken_directive, label_or_sign_text]
per_instruction_fields:
text: string
author_role: [trainer, supervisor, operator] # no personal names
task_id: string
precondition_text: string or null
exception_text: string or null
linked_media: [clip_id or null]
split_policy: held_out_by_site and by_author_role
evaluation_subset: reserved, never used in training
excluded: personal names, customer names, pay or discipline notes
rights: training and internal evaluation; no redistribution
Mixing the two
A reasonable plan, stated as a hypothesis, is to use public household sets for general language-to-action structure and workplace data for domain adaptation and evaluation. How much workplace data is enough depends on the model, the task and the site mix, and no universal figure exists. Report results on held-out sites and authors rather than on random splits of the same documents.
What this comparison cannot support
It cannot tell you how a particular model will perform on your workplace tasks; only your own held-out evaluation can. It also cannot support control training. Instructions, whether household or workplace, contain no joint states, commanded actions or contact forces, and a humanoid needs robot or teleoperation data to learn the motion itself. Footage, SOPs and recognition data on their own do not train control either.
The two public sources also have their own limits. They were not designed to represent regulated or safety-critical work, and the workplace instruction data a buyer licenses will have its own gaps in coverage and its own permissions. Instruction data written by employees may carry confidentiality or authorship questions; see who owns a procedure. This is not legal advice.
Questions to ask suppliers of workplace instruction data
- Who wrote the instructions: procedure authors, supervisors, operators, or annotators watching video?
- Are formal procedures and spoken or informal directions both present, and are they labeled by type?
- Does the text use local abbreviations, and is a glossary supplied?
- How many sites, roles and task families does the set cover, and how unevenly?
- Are instructions linked to video, procedure versions or task IDs?
- Are exceptions, preconditions and sign-offs captured, or only the nominal path?
- What names and identifiers were removed, and does that affect meaning?
- Who can authorize licensing of text written by employees or third parties?
- Can the supplier support a held-out evaluation split by site?
Discuss your requirements
If you are deciding what workplace instruction data to add beyond public benchmarks, describe the tasks, authors and permitted uses to SourceX. SourceX looks for US businesses that may hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match; SourceX reviews the requirement and follows up about potential sources. Start from the humanoid training data guide or the SOPs and procedural data hub; the catalog page on SOPs and playbooks covers the document category.
Discuss your humanoid training-data requirements with SourceX
Sources
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — household instruction benchmark; format and labeling reference only.
- BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation — everyday-activity benchmark; activity-selection reference only.
- ONET Database (ONET 29.1 per search snapshot): Task Statements and Detailed Work Activities — occupational task vocabulary for scoping.