Short answer
To evaluate instruction following in embodied agents with real procedures, convert business tasks into a fixed set of instructions at several granularities, define success per task using the procedure's own end-state criteria, hold out by site, wording or task, and score execution separately from understanding. Procedures define the tasks and rubrics; they do not supply the physical trials.
The buyer's problem
Public embodied-instruction benchmarks use household scenes, mostly in simulation. A buyer deploying to a warehouse, kitchen or hotel needs to know if the agent follows instructions as those workplaces phrase them, with their vocabulary, equipment and constraints. Real procedures provide that phrasing and the correct order of work. The difficulty is converting documents into tests that are fair, scorable and not contaminated by training.
This page is about the evaluation design. For reasoning-only tests that do not require execution, see procedural reasoning evaluation sets. For the underlying question-answering use, see procedural QA for embodied assistants.
What existing benchmarks teach about design
ALFRED maps natural-language instructions and egocentric vision to action sequences for household tasks. It pairs expert demonstrations in interactive visual scenes with 25k natural-language directives, including high-level goals and low-level step instructions (Shridhar et al., 2019). The design point is that the same task is described at two levels, so you can test whether a model needs the step-level hint.
TEACh collects more than 3,000 human-human dialogues in which a Commander guides a Follower through household tasks in simulation, and the Follower can ask questions. It defines three benchmarks: dialogue understanding, language grounding and task execution (Padmakumar et al., 2021). The relevant lesson is that clarification is part of the task, and that execution can be scored separately from understanding.
CALVIN is an open-source simulated benchmark for long-horizon language-conditioned manipulation. It tests on novel instructions, environments and objects, and reports that its MCIL baseline reaches up to 53.9% on short-horizon tasks (Mees et al., 2021). Generalization splits over instruction wording, scene and object are the transferable idea.
These are simulation or lab benchmarks, so their numbers say nothing about a deployed humanoid. Use them as design patterns.
What real procedures add, and what they cannot
| Element | What procedures supply | Gap |
|---|---|---|
| Task definitions | Authentic goals in workplace wording | Not executable without a robot, an environment and resets |
| Instruction variants | Goal-level and step-level text from the same document | Paraphrase and spoken variation need added writing |
| Success criteria | End-state statements and inspection items | Observability by your sensors; see checklists as success criteria |
| Ordering constraints | Required sequences and optional steps | Alternate valid orders that staff use in practice |
| Exceptions | Documented stop and escalate rules | Rare to trigger on demand |
A procedure alone cannot evaluate manipulation; it needs trial environments, a robot or simulator, and human or sensor scoring. If your agent only plans over text, the document set can drive a purely language-level test.
Illustrative evaluation task record
# Illustrative only - synthetic example, not an available dataset
task_id: eval_task_0213
source_procedure: sop_0117 (rev r3)
site_id: site_017
split: held_out_site
instructions:
goal: "Reset the meeting room for the next booking."
step_level: ["Collect the cups from the table", "Wipe the table", "Straighten the chairs"]
paraphrases: 3 # written by annotators
ambiguity_variant: "Tidy up in here." # tests clarification
success:
checklist_ref: chk_0042
required_items: [chk_0042_03, chk_0042_07]
forbidden_actions: [touch_personal_items]
scoring:
task_success: binary, scored by two raters
step_success: per step
asked_clarification: yes/no
safe_stop_when_blocked: yes/no
The record separates goal success, step success and behavior under ambiguity. Collapsing them into one score hides where an agent fails.
Designing the splits
Training and evaluation sets must stay separate (see procedural reasoning evaluation sets). Evaluation splits should hold out what the claim requires:
- By site or operator if you claim transfer to unseen workplaces.
- By procedure if you claim the agent follows instructions it has not seen.
- By wording if you claim robustness to phrasing; hold out paraphrase families, not just sentences.
- By time or revision if documents change; see SOP revision histories.
Documents used to build the evaluation must never appear in training prompts, retrieval indexes or fine-tuning data for the system under test. This includes near-duplicate versions; deduplicating procedures describes the check.
Granularity and wording
Step granularity changes difficulty. "Clear the table" and "pick up cup 1, place in bin" test different abilities. Decide which levels you evaluate and keep them labeled; see step granularity. Controlled vocabulary or house style in the source can make instructions sound unnatural; if you rewrite them, record that in the data documentation and compare with real vs generated procedures.
Ambiguity is a test dimension of its own. Real instructions leave out objects, locations and quantities, and a useful agent either infers or asks. Ambiguous instructions and clarification covers building those variants.
Acceptance criteria
- Scorability. Two raters agree on success for a sample; set the agreement level you need.
- Coverage. Report tasks by site, role, step count and ambiguity level.
- Leakage control. Written evidence that source documents are absent from training and retrieval data.
- Provenance. Each task cites its source procedure, revision and license terms.
- Reproducibility. The environment reset and object placement are specified well enough to rerun a trial.
Limitations and preparation
Real procedures describe the happy path; failures, blocked paths and missing items rarely appear and must be designed in. Simulation results do not transfer automatically to real hardware. Procedures from one business may over-represent its own vocabulary. And the trial environment, object sets and scoring staff are not in the documents at all. If trials use workplace footage, consent and access matter; see worker consent and identifiable people.
Questions to ask suppliers
- Which procedures can be licensed for evaluation use specifically, as opposed to training?
- Can the supplier withhold a distinct subset that they never share with other licensees?
- Do documents exist at both goal and step level, and are they linked?
- Are there real transcripts of questions workers ask about the procedure?
- Which procedures have been revised recently, and are earlier versions retained?
- Can the supplier identify equivalent procedures across sites for held-out splits?
- Are any documents available publicly or from vendors, which risks overlap with pretraining data?
Discuss your requirements
If you are designing an evaluation set, describe the tasks, sites and permitted uses to SourceX. SourceX looks for US businesses that hold suitable procedures, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. Annotation and trial design are scoped case by case, and a request does not guarantee a match; SourceX follows up about potential sources.
Start from the humanoid training data guide or the procedural data hub, or see the evaluation sets use case.
Tell SourceX what your evaluation set needs to cover
Sources
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — ALFRED pairs expert demonstrations with about 25k natural-language directives at goal and step level for household tasks.
- TEACh: Task-driven Embodied Agents that Chat — TEACh collects more than 3,000 human-human dialogues in which a Follower can ask questions while completing household tasks, with three benchmarks.
- CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — CALVIN tests long-horizon language-conditioned manipulation on novel instructions, environments and objects.