Short answer
Referring-expression annotation attaches a short phrase to one object in an image so that the phrase identifies that object and no other, for example "the left of the two gray bins". For workplace robots it teaches which item an instruction means when similar items sit together. Specify distractor-aware writing rules, box or mask targets, and verification by a second reader.
The problem it solves
Workplaces are full of lookalikes: eight identical totes on a rack, a drawer of same-size wrenches, a shelf of one product in three sizes. An instruction that names only the category ("hand me the bin") is ambiguous, and a robot needs either to ask or to resolve the reference from the scene. Referring-expression data trains and tests that resolution. Each example is an image, a target object and a phrase that picks the target out from its neighbors.
The classic research setting is general-purpose scenes. Yu et al. studied both producing and understanding such phrases on the RefCOCO, RefCOCO+ and RefCOCOg benchmarks and reported that comparing the target visually with other objects in the same image improves performance (Yu et al., 2016). The practical consequence for a buyer is that a phrase should be written against the distractors actually present, not against the object alone. A caption such as "blue box" is not a referring expression if three blue boxes are in view.
This page is the annotation specification. Where the source images come from is a separate decision; see catalog text for language grounding for product text and cluttered scenes for recognition for imagery with realistic density.
What this data can support, and what it cannot
It can support grounding of noun phrases and spatial relations to image regions, evaluation of whether a model picks the right instance, and the language-to-object stage of a larger pipeline in which a planner resolves "the one next to the scale" and passes a region to a downstream module.
It cannot teach an action. Knowing which tote the speaker means does not tell a humanoid how to lift it, regrasp it or avoid a person. Referring phrases are written from a camera's viewpoint, so "to the left" may flip when the robot looks from another side; phrases in images from a ceiling camera do not carry over to a head-mounted robot view without testing. And the language is the annotators', not necessarily your operators'; real instructions are shorter, vaguer and often spoken. Treat this data as one part of instruction-paired demonstration data, not a replacement.
Illustrative annotation specification
The specification below is Illustrative; field names and thresholds are examples to adapt.
# Illustrative referring-expression spec - synthetic values
unit: one (image, target_object, phrase) triple
image_requirements:
min_same_category_instances_in_view: 2 # otherwise a category word suffices
sources: [site_photos, licensed_catalog_in_context]
target:
geometry: instance mask preferred; box accepted if masks cost too much
object_id: stable within the image set
phrase_rules:
must_exclude_all_distractors: true
allowed_cues: [color, size, material, position, relation_to_landmark, state, text_on_label]
position_frame: image_left_right | object_centric # choose one and state it
forbidden: [viewer-dependent words unless frame stated, brand slogans, guesses about hidden parts]
length: 3-15 words
variants_per_target: 2-3 written by different annotators
verification:
second_reader_selects_target_from_phrase: required
accept_if_selected_object_matches: true
dispute_resolution: third reader; drop if still ambiguous
fields:
[image_id, object_id, phrase, annotator_id_pseudonym, frame, n_distractors,
verification_result, license_id]
How to write the rules
The rules, not the annotators, decide dataset quality. Four decisions carry most of the weight.
Choose a spatial frame. "Left" can mean left in the image, the object's own left, or the speaker's left. Pick one and give examples, or require annotators to use a landmark ("the tote below the red label") that does not depend on the frame.
Require distractor awareness. Give annotators the list of other instances of the same category in the image, or show them boxes, so they have to write something that tells the target from each. Record the number of distractors for each triple; it becomes a difficulty field you can stratify by.
Collect several phrasings per target. Different annotators write different phrasings for the same object, which exposes a model to varied vocabulary and exposes weak examples when two annotators disagree about what is salient.
Constrain length and register. Short phrases of 3 to 15 words match spoken instructions more closely than long descriptions. If your product will receive speech, ask for a sample of phrases typed as people would say them, and compare with the written set.
Acceptance and quality checks
- Second-reader test. A different person sees the image and the phrase and clicks the target. Report the share of triples where the click lands on the intended object, by distractor count.
- Ambiguity audit. Sample triples and ask whether any other object also fits. Ambiguity rates rise with distractor count, so measure at the high end.
- Frame consistency. Check that position words follow the stated frame in a sample.
- Coverage. Counts by object category, by distractor count, by site and by lighting; confirm the target objects include your difficult lookalikes, not only distinct items.
- Split hygiene. Hold out by scene or site as well as by image, so that near-duplicate frames do not appear on both sides. Training and evaluation roles are described in novel object generalization evaluation.
- Documentation. A datasheet stating who annotated, with what instructions and what the data is not for, in the form recommended by Gebru et al.
Source images and supplier types
Source images determine the ceiling. Studio packshots usually show one item, so they cannot supply distractors. Workplace photographs from inventory, quality or maintenance records can, but they carry permissions issues: people, screens, labels and third-party products may appear. Annotation can be done by the data holder, by an annotation vendor, or by a mix, with the holder checking domain terms; the right choice depends on whether annotators need domain knowledge such as fastener names. If masks are required, pair this work with segmentation masks for products, and consider whether the target must be recognized at the instance level (instance vs category recognition).
Handling ambiguity honestly
Some scenes cannot be disambiguated by any short phrase: two identical totes side by side, differing only in position. Decide what the dataset does with those. One option is to require a relational phrase ("the tote closer to the door"); another is to include a clarification label, marking that a human would need to ask which one is meant. The second is closer to how a deployed system should behave, and a dataset that never contains an unresolvable case can teach a model to guess confidently.
Also plan how to treat negatives. A useful evaluation includes phrases that match no object in the image, so the model can learn to answer "none". These are cheap to generate by pairing a phrase with a different image of the same site, and they expose models that always select something.
Limitations and extra preparation
- Annotated phrases describe how annotators see the image, which may not match how operators speak.
- A model trained on a single site's shelves may learn the layout, not the language.
- Phrases about hidden or off-screen objects need a different task design.
- Rights apply twice: to the source images and to the new annotations. Agree who owns the annotation layer and whether the images may be redistributed to you with it.
- Redaction of faces or labels in source images can remove the very cues annotators would use.
Questions to ask suppliers
- Do your images contain several instances of the same category in a single view, and how many on average?
- Who writes the phrases, what training do they receive and in what language?
- Is there a second-reader verification step, and what was its pass rate on the last batch?
- Which spatial frame do the phrases use?
- Can you deliver masks or boxes for the target and for each distractor?
- How are annotators paid and managed, and are pseudonymous IDs retained per phrase?
- Can the annotation layer be licensed separately from the images?
- What happens to disputed or ambiguous examples: are they removed or flagged?
Discuss your requirements
If you can describe the objects, the density of lookalikes and the phrase style you need, send the requirement to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. A request does not guarantee a match, and SourceX follows up about potential sources. For the broader map, see the humanoid training data guide and the object data hub.
Discuss your humanoid training-data requirements with SourceX
Sources
- Modeling Context in Referring Expressions (RefCOCO) — referring-expression generation and comprehension; benefit of comparing with other objects in the image.
- Datasheets for Datasets — documenting how a dataset was collected and annotated.