Grounding LLM Task Planners With Workplace SOPs

The planner team's sourcing problem

A language-model planner trained on web text knows how people describe making coffee. It knows far less about how an environmental-services team turns over an isolation room, how a line operator clears a jam on a case erector, or which verification a quality technician signs before releasing a batch. When such a planner fails on workplace tasks, the usual causes are missing steps, wrong order, skipped safety checks and no clear definition of done, not weak language.

Written procedures from operating businesses contain exactly that domain knowledge. The broader limits of procedural text are set out in what SOPs can and cannot teach a humanoid: procedures feed planning, grounding and evaluation, never motor skill. This page covers the planner-specific purchase, within the procedures cluster.

Five ways SOPs enter a planner

Retrieval context. In SayCan, a language model supplies task knowledge and the robot's pretrained skills filter its proposals to what is feasible in context; the system was evaluated with a mobile manipulator in a real kitchen (Ichter et al., 2022). Retrieved procedures make the task-knowledge side specific to a workplace instead of generic.

Plan exemplars. ProgPrompt prompts the model with program-like specifications of the available actions and objects plus example programs, and was deployed on a physical arm for tabletop tasks (Singh et al., 2022). Code as Policies uses few-shot prompting to turn commands into policy code that calls perception and control APIs (Liang et al., 2022). In both, the examples matter as much as the model. Real procedures, rewritten into your skill calls, give you exemplars that reflect real task structure rather than invented chores.

Constraints and checks. Inner Monologue replans from natural-language feedback, including success detection, scene description and human interaction (Huang et al., 2022). An SOP's acceptance criteria and preconditions are a ready list of the questions such feedback should answer. The field definitions belong to preconditions and postconditions.

Clarification triggers. KnowNo measures planner uncertainty with conformal prediction so the robot asks for help when it should, and reports needing 10–24% less help than baselines (Ren et al., 2023). Procedures show where trained workers are expected to stop and ask; dialogue data such as TEACh, with more than 3,000 human-human task dialogues in which the follower can ask questions (Padmakumar et al., 2021), shows how those questions are phrased. Sourcing for this capability is covered on ambiguous instructions and clarification.

Evaluation. Keep some procedure families out of every prompt and fine-tuning set and use them to score plans. Test-set design is the subject of procedural reasoning evaluation sets.

The vocabulary gap is the real preparation cost

SOP verbs are written for people: "isolate," "verify," "wipe down," "stage," "make safe." A planner can only emit calls your robot supports. Every procedure therefore needs a mapping from its steps to your skill library, and the steps that do not map are as informative as the ones that do.

The residue column is a shopping list for demonstrations and robot data. Rewriting human procedures into robot-executable steps is its own task, covered in rewriting human procedures for robots. If your planner emits PDDL or behavior trees rather than code, see procedures to PDDL and behavior trees.

Illustrative planner-ready record

# Illustrative only — synthetic procedure and skill names, not a supplier document
procedure_id: EVS_PROC_031          # isolation room turnover, site_014, rev 6
instruction: "Turn over room 4B after discharge; contact precautions."
retrieval_chunks: 14                 # one per numbered step, with step_id and revision
plan_exemplar:
  - step: 3
    sop_text: "Don gown and gloves before entry."
    skills: []                        # human-only step; planner must not emit
  - step: 7
    sop_text: "Wipe bed rails, call button and overbed table with approved disinfectant."
    skills: [fetch(wipes_canister), wipe(bed_rail_L), wipe(bed_rail_R), wipe(call_button), wipe(overbed_table)]
    precondition: "linen removed (step 6)"
    dwell_constraint: "surface stays wet for the label contact time"
    success_check: "all listed surfaces visibly wet"
  - step: 12
    sop_text: "If curtains are visibly soiled, notify the supervisor."
    clarification_flag: true          # branching step with a human decision
unmapped_residue: [curtain_soil_judgment, mattress_seam_inspection]
split: eval_holdout                   # procedure family reserved for planner testing

What SOP grounding cannot provide

• Feasibility. Whether a skill will succeed from this pose, in this clutter, on this robot comes from robot data and learned affordances, not from text.
• Perception. VoxPoser shows a language model writing code that queries a vision-language model to build value maps in the robot's observation space (Huang et al., 2023). The language side plans; the grounding still comes from perception models trained on images.
• The skills themselves. Every mapped call assumes a trained skill. The SOP provides none of the motion or force behind it.
• Implicit steps. Writers omit what every trained worker knows. Planners inherit those gaps unless someone writes them out.
• Current practice. Procedures can lag the floor. Check procedural drift before treating a document as ground truth.

Video of the same tasks can add what text lacks for planning, such as real step durations and the order people actually follow; see workplace video for task planning.

Supplier types

Structured authoring sharply reduces parsing. In S1000D, for example, procedural content is held as data modules, stand-alone units identified by a unique data module code (S1000D Steering Committee). Structured authoring exports explains what such sources look like. Whether the holder may license OEM-written content is a separate question; see who owns a procedure.

Checks before you use a corpus for planning

1. Whole procedures. Excerpts lose the preconditions and branches that make plans valid.
2. Step structure. Numbered steps and sub-steps survive extraction; prose paragraphs need rewriting.
3. Named objects and tools. Count steps that name the object acted on, not just the action.
4. Decision points. Branches and escalation steps are marked, not buried in notes.
5. Revision status. Only current, approved versions go into retrieval; superseded ones are labeled.
6. Split integrity. Near-identical procedures across sites stay on one side of the train/test line.

The full pass/fail list is the SOP corpus evaluation checklist.

Questions to ask suppliers

1. Which tasks and equipment do the procedures cover, and how many distinct procedure families are there?
2. Are procedures authored in a structured system, a platform, word-processor files or scans?
3. Do steps name the objects, tools, locations and parameters involved?
4. Where do procedures tell workers to stop, escalate or ask a supervisor?
5. Are safety cautions and interlocks written as separate, marked elements?
6. Which revision is current for each document, and are older revisions retained?
7. Are completed checklists or platform completion records available for the same procedures?
8. Were any procedures adapted from equipment makers, consultants or franchisors?

Discuss your requirements

If your planner needs domain procedures, describe the task families, equipment and planner interface to SourceX. SourceX looks for US businesses that hold suitable procedures, checks the data and the supplier's licensing permissions, and coordinates pricing, allowed uses and delivery in a license. It reviews the request and follows up about potential sources; a request does not guarantee a match. For other data types, start from the humanoid training data guide. The catalog entry for SOPs, playbooks and internal knowledge bases and the robotics and embodied AI use case give wider context.

Explore sourcing and licensing options with SourceX

Sources

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances — language task knowledge filtered by robot skill feasibility.
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models — prompts listing available actions and objects with example programs.
• Code as Policies: Language Model Programs for Embodied Control — few-shot policy code calling perception and control APIs.
• Inner Monologue: Embodied Reasoning through Planning with Language Models — replanning from success detection and scene feedback.
• Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners — planners that ask for help when uncertain.
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models — language plans grounded through vision models.
• TEACh: Task-driven Embodied Agents that Chat — task dialogues in which the follower asks questions.
• S1000D International specification for technical publications using a common source database — procedures held as uniquely identified data modules.