Short answer
Natural worker instructions are the short, elliptical, often spoken phrasings people use when telling a colleague what to do, such as "grab the blue one and flip it". They differ from SOP language and need purposeful collection: show workers a scene or task, record what they say, and pair each utterance with the task, objects and context. They support language grounding, not robot control.
The buyer's problem
Procedures are written for audit and training. People at work talk differently. A supervisor says "swing that cart over by the dock", not "Move cart C-14 to staging position 3". Models trained only on document language or on templated instructions often meet a different register when deployed beside real staff. A buyer who wants a robot to follow spoken or typed requests from workers needs examples of how those workers actually phrase them.
This data rarely exists in usable form. Chat logs and call recordings contain instructions, but without the scene they refer to. Purpose-built collection can pair language with context. The cost is design effort, recruitment and consent, which is why this page is a specification rather than a sourcing hunt. For how documents can ground language, see procedural text for instruction grounding.
What published datasets show about pairing
Public benchmarks illustrate the pairing idea. ALFRED pairs expert demonstrations with 25k natural-language directives, covering high-level goals and low-level step instructions, for household tasks in interactive visual scenes (Shridhar et al., 2019). TEACh collected more than 3,000 human-human dialogues in simulation in which a Commander guides a Follower, who can ask questions (Padmakumar et al., 2021). BridgeData V2 offers 60,096 robot trajectories across 24 environments and is compatible with goal-image and natural-language conditioned methods (Walke et al., 2023).
None of these is workplace speech from workers in a given trade, and the first two are simulated household settings. They show the shape of the deliverable: an utterance tied to an episode, a goal level and a scene. They do not tell a buyer how many instructions a given workplace model needs, and nothing here implies a minimum.
Elicitation protocols
The protocol determines what language you get. Common designs:
| Protocol | What the worker sees | What it yields | Main risk |
|---|---|---|---|
| Scene-to-instruction | Photo or live scene with a target change | Direct commands matched to a visible goal | Over-describes; reads like captions |
| Before/after pair | Start and end states | Goal-level requests | Misses step-level language |
| Video-to-instruction | A clip of the task | Both goal and step phrasing | Narration drifts into description |
| Role-play handoff | A colleague who acts on the words | Real ambiguity and corrections | Needs a human actor, higher cost |
| Dialogue with clarification | Two participants, one with knowledge | Questions and answers, as in the TEACh pattern | Harder to label and scale |
| Free request | A task goal in a job role | Varied, natural register | Weak grounding to a scene |
Pick the protocol that matches the deployed interaction. Hands-free voice input differs from typed tickets, and both differ from a worker pointing while speaking. If pointing or gaze will matter, capture it; text alone will not recover it.
Illustrative collection specification
# Illustrative - all values synthetic
collection: natural_instructions_spec_v0
task_families: [restock_shelf, move_cart, clean_surface] # buyer-defined
modalities: [audio_16kHz_mono, transcript, scene_image]
elicitation: scene_to_instruction + role_play_handoff
speakers:
count_target: "buyer-defined"
stratify_by: [trade, experience_band, first_language, region]
instruction_levels: [goal, step, correction]
per_item_fields:
item_id: ins_000412
speaker_id: spk_A12 # pseudonymous
task_id: restock_shelf
scene_ref: img_0098
utterance_audio: aud_000412.wav
transcript_verbatim: "put the cans, no, the soup, up on the second shelf"
transcript_normalized: "put the soup cans on the second shelf"
level: goal
referents: [{span: "the soup", object_id: obj_031}]
disfluencies_kept: true
ambiguity_flag: false
executed_by: human_actor # human_actor | none
outcome: success # success | failed | clarified
consent:
recording: signed
derived_use: "model training, evaluation"
voice_retention_policy: documented
Two details deserve attention. Keep the verbatim transcript beside any normalized one, because disfluency and self-correction are part of what a robot must handle. And store referents, spans in the utterance tied to objects in the scene, since an instruction without referents cannot be graded for grounding.
Diversity dimensions to specify
Instruction diversity is more than paraphrase count. Specify coverage along:
- Speakers. Trade, tenure, accent, first language, and speaking style under noise.
- Register. Command, request, hint ("it's getting full"), question and correction.
- Granularity. Goal, subgoal and single-step phrasing; see step granularity.
- Reference style. Names, attributes, deictics ("that one"), spatial terms and local jargon, which the workplace terminology glossaries page covers.
- Ambiguity. Underspecified or conflicting requests and the clarifying exchange, described on ambiguous instructions and clarification.
- Environment. Scenes with and without clutter, noise levels, and multiple similar objects.
Acceptance tests
- Referent audit. Sample items and check that each referent resolves to exactly one object in the paired scene.
- Transcript accuracy. Compare verbatim transcripts with audio on a random sample; the error threshold is set by the buyer.
- Speaker spread. Report items per speaker, so one fluent talker does not dominate.
- Duplicate and template check. Count near-identical utterances; a high rate suggests priming from the task script.
- Grounding test. Give a second person only the utterance and scene and ask them to act; the share who succeed is a usable measure of whether the instruction was adequate.
- Held-out split. For evaluation, hold out whole speakers or sites, not random utterances, so voices and phrasings do not leak. This follows the training-versus-evaluation distinction in instruction-following evaluation for embodied agents.
What this data cannot do, and extra preparation
- No actions. Instructions are language. To train a policy, they must be paired with robot or demonstration data; see instruction-paired demonstration datasets.
- Human listener, not robot. A person who knows a colleague can fill gaps speaks differently than to a machine. A role-play handoff with a person acting is not the same as speaking to a robot, and some workers adapt their phrasing once they know a system is listening.
- Elicited, not spontaneous. Phrasing prompted by a scene is closer to real than document text but still staged.
- Simulation to workplace gap. Public benchmarks are largely simulated household scenes; workplace transfer is untested by them.
- Language coverage. One language or accent set limits who the model serves.
- Rewriting. If workers' phrasing must be mapped to robot-executable steps, that is a separate task covered under rewriting human procedures for robots.
Permissions and provenance
Voice is personal data in many jurisdictions and can identify a speaker. Consent must cover recording, transcription, model training, evaluation and any retention of raw audio, and workers should be able to understand it. Employer-run collection raises questions about whether participation is truly voluntary. The content of instructions may also reveal confidential processes or customer details. This is not legal advice. See worker consent and identifiable people.
Questions to ask collection providers
- Which elicitation protocol will be used, and why does it match the deployed interaction?
- How are speakers recruited and stratified, and how many items per speaker are capped?
- Are verbatim and normalized transcripts both delivered?
- How are referents annotated and verified?
- How are prompt scripts prevented from priming the wording?
- What consent covers voice, and can a speaker withdraw?
- Are held-out speakers or sites reserved for evaluation?
- What is the recording environment, microphone type and noise level?
Handling clarification and correction turns
Real instruction use includes turns where the first request fails. A worker says "bring me the tote", the listener asks "which one?", and the worker points or adds a color. A collection that records only the first utterance loses the repair. If the deployed system will ask clarifying questions, specify that sessions capture the full exchange, label each turn as request, question, answer or correction, and record whether the task was completed afterward. Keep the original ordering and timing, because overlap and interruption are part of how people actually talk.
Corrections after a mistake are a second source of signal. When the actor picks the wrong item and the worker says "no, the other one", the pair of utterance and error reveals which descriptions were insufficient. Capturing these requires a human actor or a robot making real mistakes, which raises cost, so decide in advance whether the target use justifies it.
Discuss your requirements
If you need worker-phrased instructions for a defined set of tasks, describe the trades, tasks, languages and permitted uses to SourceX. SourceX reviews the requirement and follows up about potential sources; if no existing asset fits, SourceX can scope a commissioned collection. Annotation and specialist capture partners can be arranged case by case, and a request does not guarantee a match. Start from the humanoid training data guide or the procedures hub for related material.
Describe the workplace demonstration data your robotics project needs.
Sources
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — 25k natural-language directives paired with expert demonstrations.
- TEACh: Task-driven Embodied Agents that Chat — more than 3,000 human-human dialogues with clarifying questions.
- BridgeData V2: A Dataset for Robot Learning at Scale — 60,096 trajectories, language and goal-image conditioning.