Procedural Text for Instruction Grounding in Vision-Language-Action Models

What grounding needs that robot datasets may not have

A vision-language-action model maps an instruction and an observation to an action. RT-2 introduced the class by co-training a vision-language model on robot trajectories alongside web-scale vision-language tasks, with robot actions expressed as text tokens (Brohan et al., 2023). Robot datasets built for this attach language to episodes: DROID records language instructions per episode (Khazatsky et al., 2024), and BridgeData V2 supports natural-language conditioned methods (Walke et al., 2023).

Those instructions describe the tasks each dataset was collected on. Workplace instructions look different. They name parts and fixtures ("the dancer arm", "bay 3, shelf 4"), carry quantities and parameters, attach conditions ("if the seal is torn, set the unit aside"), point to standards ("as in photo P-7"), and bundle several actions into one sentence. A model that has never seen this kind of language has to generalize to it at deployment, which is a risky place to discover a gap.

Procedures written by operating businesses are a large existing body of that language. They are the source this page is about. Text attached to demonstration video, such as captions and narration, is specified separately on language annotations for demonstrations.

Five ways procedural text enters a VLA program

1. The instruction distribution for robot episodes. When you collect or license robot episodes for workplace tasks, procedure steps give the vocabulary and phrasing for each episode's instruction and its paraphrases. This keeps training language close to deployment language.

2. A step library for hierarchical control. A high-level model can emit step instructions that a low-level policy executes. SayCan showed the pattern: a language model proposes steps, and the robot's skill affordances filter them to what is feasible, roughly halving errors against non-grounded baselines (Ichter et al., 2022). Procedures supply the step library; planner-side use is covered on grounding LLM planners with SOPs.

3. Held-out instruction sets. CALVIN evaluates language-conditioned policies on novel instructions, environments and objects (Mees et al., 2021). Real procedure wording from sites your model has never seen makes a demanding held-out set; instruction-following evaluation for embodied agents covers the protocol.

4. Two-level pairing with demonstrations. In ALFRED, expert demonstrations come with 25k natural-language directives at two levels, high-level goals and low-level step instructions (Shridhar et al., 2019). A workplace procedure has the same shape, a task goal over ordered steps, so it can label demonstrations at both levels once the video exists.

5. Ambiguity and clarification. Real instructions are often underspecified. KnowNo aligns a planner's uncertainty so the robot asks for help when needed, reducing the help required by 10–24% compared with baselines (Ren et al., 2023). Ambiguous procedure steps are natural test items for that behavior.

What to ask suppliers for

• Step-level SOPs and work instructions in their current revision, with revision history.
• Tool, part and material lists, plus any site glossary, so object names can be grounded consistently.
• Photos or diagrams per step where they exist, which let nouns be grounded in images as well as text.
• Instructions as people actually give them: task assignments, shift handover notes and work-order descriptions. Formal procedures and spoken instructions differ; natural worker instructions covers collecting the second kind.
• Other languages used at the site, if your deployment is multilingual.

Typical holders are operating businesses with documented operations, users of connected-worker platforms, and franchisors with operations manuals. Annotation vendors usually do the preparation below. The catalog page on licensing SOPs and playbooks describes the underlying asset, and the robotics and embodied AI use case places it among other robotics data.

Turning procedure text into grounding records

Raw procedure steps are written for trained people. Before they can condition or test a policy, each step needs preparation:

1. Split compound steps so each record holds one action.
2. Resolve references such as "it", "the above" and "as before".
3. Separate visible from non-visible content. "The blue gasket" is perceivable; "the revision-B gasket" may not be. Keep the second in a note.
4. Tag slots: verb, objects, source, target, condition and parameter.
5. Map to the robot's skills, and flag steps with no matching skill as data gaps.
6. Write paraphrases with several annotators, keeping the original wording as a separate field.
7. Keep provenance: document, revision and site for every record.

Rewriting steps into robot-executable form is its own topic, covered on rewriting human procedures for robots; the underlying field set follows the procedure step schema.

Illustrative package: grounding records from one step

# Illustrative only — synthetic records derived from one work-instruction step
source: {doc_id: WI-ASM-212, revision: E, step: 6, site: site_031}
original_text: "Seat the gasket in the groove with the tab at 12 o'clock, then fit the cover and hand-start all four screws."
records:
  - id: gr_000881
    instruction: "Place the gasket in the groove with its tab at the top."
    slots: {verb: place, object: gasket_black_round, target: housing_groove, constraint: "tab at 12 o'clock"}
    robot_skill: place_deformable_ring      # present in the skill library
    paired_episode: teleop_ep_04417          # robot episode exists
    paraphrases: ["Fit the gasket into the groove, tab facing up.",
                  "Put the round gasket in the housing groove so the tab points up."]
  - id: gr_000882
    instruction: "Fit the cover over the gasket."
    slots: {verb: fit, object: cover_plate, target: housing}
    robot_skill: place_rigid
    paired_episode: null                     # no episode yet: collection gap
  - id: gr_000883
    instruction: "Hand-start the four cover screws."
    slots: {verb: hand_start, object: screw_m4, count: 4}
    robot_skill: null                        # no skill: flagged for teleoperation collection
non_visual_notes: ["gasket part revision appears only on the bill of materials"]

The two null fields are the point: they show where text has outrun the robot data that would ground it.

What the text can and cannot support

Can support: workplace vocabulary and phrasing in instruction-conditioned training; step decomposition for hierarchical policies; held-out instruction sets for generalization tests; noun grounding in images when steps carry photos; ambiguity and clarification tests.

Cannot support: the mapping from words to motion. Without robot episodes or demonstrations of each step, a model learns to parse workplace language without being able to act on it. Procedures also leave out what trained workers already know, so some instructions will be underspecified for a robot. The keystone page on what SOPs can and cannot teach a humanoid explains the boundary.

Quality checks before acceptance

• One action per record, checked on a random sample.
• Visible referents: every object slot names something a camera can see, or carries a note.
• Paraphrase diversity: paraphrases differ in wording, not only word order; the threshold is yours.
• Skill coverage: share of records mapped to an existing robot skill, reported per task.
• Provenance complete: document, revision and site on every record.
• Held-out split: evaluation records separated by site or company from training records, with no shared procedures.

Questions to ask suppliers

1. Which tasks and sites do the procedures cover, and how current are they?
2. Are steps numbered and structured, or written as paragraphs?
3. Do steps include photos, part numbers or tool lists that identify objects?
4. Are there task assignments, handover notes or work orders that show how instructions are given in practice?
5. In which languages are procedures written, and are translations maintained?
6. Which documents came from equipment makers or consultants rather than your own staff?
7. Can the same tasks be recorded on video or with a robot, so instructions can be paired with episodes?

Discuss your requirements

If you need workplace instruction data for a VLA program, describe the tasks, languages and pairing you need to SourceX. SourceX looks for US businesses that hold suitable procedures, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. It reviews each requirement and follows up about potential sources; a request does not guarantee a match. Other procedure assets are listed on the procedures hub, and the humanoid training data guide covers the full section.

Explore sourcing and licensing options with SourceX

Sources

• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control — the VLA model class and its training mix.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — per-episode language instructions in robot data.
• BridgeData V2: A Dataset for Robot Learning at Scale — robot data for language-conditioned methods.
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances — language step proposals grounded by robot skills.
• CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks — evaluation on novel instructions.
• ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks — goal- and step-level instructions paired with demonstrations.
• Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners — asking for help under ambiguous instructions.