Short answer
Procedural question-answer pairs ask things like which step comes next, what tool a step needs or what to do if a check fails, with answers tied to the source procedure. They suit assistants that talk to workers and evaluation of procedural comprehension. They do not train manipulation, and a correct answer does not mean a robot can perform the step.
Who needs this and why
Not every embodied system is meant to act alone. A humanoid or wearable assistant working beside a technician may spend much of its time answering: which torque sequence, which panel, what comes after the seal check. A buyer building or evaluating that behavior needs questions that look like what workers ask, and answers a reviewer can trace to a document.
This is a text-grounding product. It sits with procedural reasoning evaluation sets and grounding LLM planners with SOPs, and it should be bought with that purpose in mind rather than as a general language corpus.
Existing material versus commissioned pairs
Existing material. Businesses already hold question-and-answer records: internal help desks, supervisor chat threads, training-course quizzes, forum posts with accepted answers, knowledge-base articles and support tickets. The TechQA dataset shows the pattern in technical support: it built question-answer pairs from forum questions whose accepted answers pointed to a companion corpus of technical notes, giving 600 training, 310 development and 490 evaluation pairs against 801,998 notes (Castelli et al., 2019). The lesson for buyers is the structure: a question, an answer and a retrievable source document, with the evaluation set kept apart. Note how small the labeled sets were against the document corpus; the value sat in the linkage, not in volume.
Existing archives often answer questions about products and tickets rather than about physical procedures, so check how much is genuinely procedural. See customer support tickets and document AI and RAG data for the adjacent supply.
Commissioned pairs. Annotators read a procedure and write questions at defined difficulty levels, then a second reviewer verifies each answer against the text. This allows control over step coverage, but pairs written by annotators drift toward wording that echoes the document, which makes them easier than real worker questions. Mix in real questions from the business where possible.
What kinds of questions to specify
- Sequencing: what happens immediately after step 6; what must be finished before step 9.
- Resources: which tool, material or personal protective item a step requires.
- Conditions: what to do if a reading is outside range; when a step may be skipped.
- Rationale: why a step exists, where the document states it.
- Location and identification: where on the equipment a part sits, where the document includes a figure or label.
- Unanswerable: questions the document does not answer, with the expected response being a refusal or an escalation.
The last category is the one buyers forget. A model trained only on answerable questions learns to invent plausible steps, which is dangerous in procedures with safety-critical content.
Illustrative record format
The record below is Illustrative; every value is synthetic. Field names are suggestions to adapt to your pipeline and to a step schema.
{
"qa_id": "qa_site_014_00231",
"doc_id": "SOP-site_014-0107",
"doc_version": "B",
"question": "After the leak check, what is the next step?",
"question_origin": "worker_chat | annotator_written",
"question_type": "sequencing",
"answer": "Step 8: record the reading on the form and tag the unit.",
"supporting_spans": [{"step": "7"}, {"step": "8"}],
"answerable": true,
"needs_figure": false,
"reviewer_agreement": "2_of_2",
"split": "eval_holdout_site"
}
Where this overlaps its neighbors
Three adjacent needs are easy to blur with QA pairs. If the point is to test whether a model asks a question when the instruction is underspecified, that is ambiguous instructions and clarification data. If the answers are produced by a model reading the procedure and you need to check them, the work is validating LLM-extracted steps. If the target is whether a robot does what a worker says, see instruction-following evaluation for embodied systems. QA over documents is the narrow case where the output is an answer, not an action.
Quality and acceptance criteria
- Every answerable item points to specific steps or spans, not just a document.
- A sample of items is re-answered by a second reviewer who has not seen the first answer, and disagreements are reported.
- Questions that depend on a figure or table are flagged, since text-only models cannot answer them; parsing scanned manuals explains why such content is often lost.
- Real and annotator-written questions are labeled, so you can measure the gap.
- Evaluation items are held out by document family or site; paraphrases of the same question must not straddle the split.
What this data cannot support
QA pairs measure whether a system can read and recall a procedure. They do not measure whether it can carry out a step, and they cannot teach grasping, force or navigation. A humanoid's control policy needs robot or teleoperation data, and QA text over SOPs cannot substitute for it.
Two further limits matter. First, a document can be wrong or out of date, and a correct answer to it is not a correct answer about the floor; see procedural drift. Second, questions asked in a chat window differ from those asked at a workstation. Work on assistants for physical tasks, such as the egocentric human-interaction dataset HoloAssist, captures the interaction as it happens, which text archives cannot reproduce. If your assistant must respond to what the worker is doing, you need that kind of capture, not only text pairs.
Permissions and provenance
Help-desk and chat-derived questions are written by employees and may include names, device identifiers and incident details. Documents behind the answers may come from equipment makers or customers. Redaction changes the data but does not by itself settle who may license it. Ask who wrote each question, whether a worker's message can be licensed, and whether the underlying manual is the supplier's to license.
Questions to ask suppliers
- Are the questions real worker questions, annotator-written, or generated by a model and filtered?
- How is each answer tied to a specific step or span of a specific document version?
- What share of items are intentionally unanswerable?
- Who reviewed answers, and how were disagreements resolved?
- Do items depend on figures, tables or part diagrams, and are those included?
- Are conversation threads kept intact, or only isolated question-answer pairs?
- Which documents are authored by third parties, and can the supplier license questions about them?
- Can a sample of 20 items be shared with source passages before you scope the full set?
Discuss your requirements
If you need procedure-grounded QA for an assistive embodied system, describe the procedures, question types, document sources and rights through a buyer request. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match. SourceX can arrange annotation, scoped case by case. For orientation, see the humanoid training-data section and the procedures hub.
Discuss your humanoid training-data requirements with SourceX.
Sources
- The TechQA Dataset — technical-support QA pairs linked to a companion note corpus, with split sizes.
- HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World — egocentric interaction data for assistants, cited for existence and nature.