Short answer
Ambiguous-instruction data pairs an underspecified request with the scene or procedure context that makes it ambiguous, the clarifying question a competent person would ask, and the answer that resolves it. It can support evaluating when an embodied planner should ask for help. It is mostly written by annotators, needs scene context to be valid, and does not train the physical act itself.
The capability being tested
A worker tells a helper "bring me the wrench." If three wrenches are on the bench, the right move is to ask which one. If one is, asking is a nuisance. Embodied agents need to tell those cases apart, and the decision depends as much on the scene as on the words.
Published planning work treats this as a calibration problem. KnowNo measures and aligns the uncertainty of a language-model planner so the robot asks for help when it needs to, using conformal prediction to give statistical task-success guarantees with limited human help; the authors report reducing the help needed by 10 to 24 percent against baselines (Ren et al., 2023). The method needs ambiguous cases to calibrate and test on. Where those come from is a data-sourcing question, and it is why this page exists.
For the neighboring cases, see procedural QA for embodied assistants (the assistant answers) and natural worker instructions (the instruction as workers phrase it).
Types of ambiguity worth labeling
| Type | Example (synthetic) | What resolves it |
|---|---|---|
| Referential | "Move the box" with several boxes | Which one, by position or attribute |
| Quantity | "Bring some filters" | How many |
| Location | "Put it on the shelf" | Which shelf or slot |
| Preference | "Use the usual tool" | The worker's usual choice |
| Procedural | "Finish the checks" where the SOP has variants | Which checklist applies |
| Safety | "Clear the area" with a hazard unknown to the agent | What is hazardous here |
| Not ambiguous | "Put the red bin on the top shelf" | Control case, no question needed |
The last row is critical. Datasets in which every instruction is ambiguous teach a system to ask about everything. Include unambiguous controls in the proportion the buyer sets, along with cases where the answer can be inferred from the procedure and so no question is warranted.
What existing dialogue data shows
Human-human dialogue about embodied tasks exists as research data. TEACh collected more than 3,000 human-human dialogues in which a Commander guides a Follower completing household tasks in simulation, with the Follower able to ask questions; it defines benchmarks for dialogue understanding, language grounding and task execution (Padmakumar et al., 2021). It is a research resource in a household simulator, so it shows the format of the problem rather than workplace vocabulary or conditions. A buyer targeting workplaces needs equivalent material in that setting.
The mirror image is a human correcting a robot after the fact, which is a different signal and is covered in trainer feedback and corrections. Here the agent asks before acting.
Real workplace material exists inside businesses as supervisor-worker message threads, radio transcripts, help-desk chats and dispatcher logs. These contain real clarifications, but the scene is usually not recorded, so you cannot verify whether the ambiguity was genuine. See workplace messaging and chat data for the adjacent supply.
Building the data: annotators and procedures
The commonest design is to start from a licensed procedure or a scene description and have annotators write instructions at controlled ambiguity levels, then write the clarifying question and the resolving answer, and have a second reviewer judge whether the question was needed. The procedure supplies realistic objects and constraints; the annotators supply the language. This is where annotation scoping enters, and a buyer should state the ambiguity types, the control ratio and the review rules. Step granularity matters because instructions pitched at a coarse level are ambiguous about steps that a fine-grained SOP defines.
Illustrative record
The record is Illustrative; all values are synthetic.
{
"item_id": "amb_site_009_0140",
"context": {
"type": "scene_description",
"text": "Bench holds two torque wrenches (blue handle, red handle) and one ratchet.",
"procedure_ref": "SOP-site_009-0058/step_4"
},
"instruction": "Hand me the wrench.",
"ambiguity_type": "referential",
"needs_clarification": true,
"clarifying_question": "Do you want the blue-handled or the red-handled torque wrench?",
"resolving_answer": "The red one.",
"resolved_instruction": "Hand me the red-handled torque wrench.",
"reviewer_votes": {"needs_question": 2, "no_question_needed": 0},
"origin": "annotator_written"
}
Quality and acceptance criteria
- Each item has a scene or procedure context sufficient to judge ambiguity; items without one are excluded or marked.
- Reviewer agreement on "question needed" is measured and reported by the supplier, with disputed items kept visible.
- Controls are present and labeled, including some that look ambiguous but are resolved by context.
- Clarifying questions are specific. "What do you mean?" is not an acceptable question when a targeted one is available.
- Evaluation items are held out by procedure family and annotator, not randomly, to prevent a model learning an annotator's habits.
What this data cannot support
Ambiguity datasets test decisions about when to ask. They cannot teach manipulation, navigation or force control, and they do not show a robot acting on the resolved instruction. A humanoid's control policy needs robot or teleoperation data, and instruction text alone cannot train it. See instruction-paired demonstration datasets for paired action data, and instruction-following evaluation for testing the whole chain.
Limits specific to this data:
- Annotator realism. Written ambiguity often differs from the ambiguity workers produce under time pressure.
- Scene text is not the scene. A description can omit what a camera would show, so a model may decide on cues the real robot lacks.
- Calibration is not supplied. The data labels cases; whether an agent's uncertainty is calibrated is a property of the model and method.
- Cost of asking. Human tolerance for questions depends on the workplace and is not in the text.
Permissions and provenance
Real message threads carry employee names, equipment details and sometimes customer references. Annotator-written items derived from a company's procedures inherit the procedure's license position; a supplier who licensed a manual may not have granted derivative use. Ask both. Redacting names does not by itself settle either point.
Questions to ask suppliers
- Are instructions written by annotators, taken from real messages, or generated by a model and filtered?
- What context accompanies each item: a procedure step, a scene description, an image or video?
- How many reviewers judged "needs clarification", and what was the agreement?
- What share of items are unambiguous controls?
- What is the ambiguity taxonomy, and who defined it?
- Do clarifying answers vary, or is there one canonical answer per item?
- Are source procedures licensed for derivative text, and by whom?
- Can a sample of 20 items with context be reviewed before you scope the full set?
Discuss your requirements
If clarification data belongs in your evaluation or training plan, describe the settings, ambiguity types, context modality and review rules through a buyer request. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. Annotation and any commissioned writing are scoped case by case. See the procedures hub and the humanoid training-data section for context.
Tell SourceX what your evaluation set needs to cover.
Sources
- Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners — KnowNo; asking for help when planner uncertainty warrants it.
- TEACh: Task-driven Embodied Agents that Chat — human-human dialogues for embodied household tasks with questions.