Procedural Reasoning Evaluation Sets Built From Workplace Procedures

Where this evaluation fits

Three evaluations are easy to confuse. Procedural reasoning, the subject here, gives a model text or images and asks questions about the procedure. Task understanding asks a model to interpret video of the work; see task understanding evaluation sets. Instruction following asks an embodied agent to carry out instructions; see instruction-following evaluation for embodied agents.

Procedural reasoning is the cheapest of the three to run, and it catches a class of failure before any robot time is spent. A planner that orders steps wrongly, forgets a precondition or misses a safety step will fail on hardware for reasons a text test could have shown.

Item types

Ordering items must accept every valid order, not only the documented one. That requires the dependency structure described on task graphs from procedures; without it, a model that proposes a valid alternative is marked wrong.

What published benchmarks show

Video-based procedure benchmarks already test several of these reasoning types. Ego4D Goal-Step's tasks include goal inference, step prediction, hierarchical relation learning and long-term temporal modeling (Song et al., 2023). CaptainCook4D benchmarks error recognition, multistep localization and procedure learning over normal and intentionally erroneous recipe executions (Peddi et al., 2023). COIN spans 180 tasks in 12 domains with hierarchical step annotations (Tang et al., 2019).

Domain-specific question sets do not need to be large to be useful. TechQA built technical-support questions from real forum posts whose accepted answers sit in vendor technical notes, with 600 training, 310 development and 490 evaluation pairs (Castelli et al., 2019). The lesson for procedure evaluation is that real questions grounded in real documents test what a model will face; volume matters less than coverage and correctness.

Many public procedural benchmarks cover cooking, toy assembly, everyday activities or software support. Service, logistics and industrial procedures are thinly represented, which is the gap workplace suppliers can fill.

Why real procedures, not generated ones

Language models can write plausible procedures on demand, and items built from them are cheap. They are also generic: they rarely contain the site-specific constraints, unusual tool choices and partial orders that make real procedures hard. The comparison is covered on real vs generated procedures.

Private procedures also help with benchmark contamination. Procedures published online, including many equipment manuals, may already sit in pretraining corpora. Internal documents held by operating businesses are less likely to, though that is something to check, not assume.

Building the set

1. Sample procedures across tasks, sites and domains, recording document ID and revision for each.
2. Fix valid orders with a task graph or expert confirmation.
3. Draft items from templates or with a model, recording the method for each item.
4. Write gold answers with acceptable alternatives and a short justification.
5. Verify independently with a second domain expert; mark or drop items they disagree on.
6. Test answerability: if an item can be answered by common sense without the procedure, it does not test domain knowledge.
7. Hold out by company, site or domain, as your claim requires, and keep the set out of every training pipeline; see held-out data.
8. Document the set. Datasheets record a dataset's motivation, composition, collection process and recommended uses (Gebru et al., 2018); the glossary entry on datasheets for datasets summarizes the idea.

Illustrative items

[
  {
    "_comment": "Illustrative only — synthetic items built from a synthetic procedure",
    "item_id": "pr_000311",
    "type": "step_ordering",
    "source": {"doc_id": "WI-ASM-212", "revision": "E", "site": "site_031"},
    "input": ["Fit the cover plate.", "Clean the housing groove.", "Stage four M4 screws.",
              "Seat the gasket, tab at 12 o'clock.", "Inspect the housing.", "Hand-start the screws."],
    "gold": {"valid_orders_from_graph": "tg_WI-ASM-212_revE", "count_valid_orders": 5},
    "verified_by": ["sme_A", "sme_B"]
  },
  {
    "item_id": "pr_000312",
    "type": "precondition_check",
    "input": {"state": "Cover plate removed; gasket lying on the bench.", "proposed_step": "Hand-start the screws."},
    "gold": {"may_start": false, "missing": ["gasket seated", "cover fitted"]},
    "verified_by": ["sme_A", "sme_C"]
  },
  {
    "item_id": "pr_000313",
    "type": "missing_step",
    "input": "Procedure with step 2 removed",
    "gold": {"missing_step": "Clean the housing groove.", "position": 2,
             "why_it_matters": "debris under the gasket causes leaks"},
    "verified_by": ["sme_B", "sme_C"]
  }
]

Licensing an evaluation set

An evaluation set reveals the procedures it was built from, so licensing terms matter more than usual. Ask for evaluation-only use, confidentiality for item text, control over any publication of results that quote items, and clarity on who owns items written by your experts from the supplier's documents. Keep evaluation licenses separate from training licenses; training vs evaluation datasets explains why mixing them undermines both. Ownership of the underlying documents is covered on who owns a procedure.

What the set can and cannot measure

Can measure: knowledge of order, dependencies, preconditions, tools, safety controls and finished states; planner quality across domains; regressions between model versions.

Cannot measure: whether a robot can perform any step, whether its cameras can perceive the states named, or whether the procedure matches current practice. Gold answers reflect the document; if the document has drifted from the floor, so have the answers.

Supplier types

• Operating businesses with documented procedures across many tasks and sites.
• Training organizations and corporate learning teams with existing quizzes and assessment items; see LMS training courses.
• Domain experts and annotation vendors who write and verify items under the buyer's guideline.

Questions to ask suppliers

1. Which tasks, sites and domains do your procedures cover, and how current are they?
2. Are any of these procedures published or shared outside the company?
3. Do you hold deviation reports, incident summaries or assessment questions tied to the procedures?
4. Can a supervisor or engineer confirm which step orders are acceptable?
5. Do existing quizzes or practical assessments test procedure knowledge, and who wrote them?
6. Will you permit evaluation-only use with confidentiality for item text?
7. Who owns items that our experts write from your documents?

Discuss your requirements

If you are building a procedural reasoning benchmark, tell SourceX which domains, item types and hold-out rules you need. SourceX looks for US businesses that hold suitable procedures, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX's role is sourcing, rights review, licensing and delivery coordination; item writing is scoped case by case. It reviews each requirement and follows up about potential sources; a request does not guarantee a match. See the procedures hub, the humanoid training data guide and the catalog use case for private evaluation sets.

Tell SourceX what your evaluation set needs to cover

Sources

• Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities — goal inference and step prediction tasks.
• CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities — error recognition and procedure learning benchmarks.
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis — hierarchical step annotations across many domains.
• The TechQA Dataset — a small, real, domain-specific QA set grounded in enterprise documents.
• Datasheets for Datasets — documenting a dataset's composition and intended uses.