Short answer
Contract annotation as its own service with its own acceptance test. The statement of work should fix the label schema and guideline version, a gold set and agreement thresholds the buyer sets, audit sampling, data-handling rules, who owns the labels, and what happens on failure. Annotation vendors can add labels to data but cannot grant rights in the underlying footage or images.
This page covers contract topics and acceptance mechanics. It is not legal advice.
The buyer's problem
Raw video, images or text become training data only after labels are added: step boundaries, hand-object interactions, segmentation masks, defect classes, captions. The buyer often has a data supplier and a separate annotation provider, sometimes the same company, sometimes not. Problems arise when the annotation scope is described as "label the data" and quality is discovered afterward.
Acceptance criteria turn "good enough" into tests that both sides can run. They also protect the supplier from open-ended revisions. Because rights, quality and confidentiality are separate issues, this page treats annotation as a service layered on top of licensed data.
What annotation can and cannot do
Annotation can add structure: segments, tracks, masks, poses, captions, classifications and quality flags. It can standardize vocabulary across sites. It cannot create information the data lacks. Hands that are out of frame cannot be labeled reliably; actions and forces absent from human video cannot be annotated into existence; blurred regions cannot be un-blurred. See annotation quality for task video for the limits specific to video.
Annotation also does not transfer rights. An annotation vendor works on data it receives under the buyer's or supplier's license. The vendor's agreement should treat the underlying data as confidential, restrict reuse, and make clear that the vendor acquires nothing in it.
Elements of the statement of work
| Element | Specify | Why |
|---|---|---|
| Scope | Data units, modalities, label types, languages | Prevents scope drift |
| Schema | Classes, fields, formats, version | Labels cannot be recovered if the schema changes |
| Guidelines | Document with definitions, examples, edge cases | The standard annotators are held to |
| Gold set | Items labeled by experts, hidden from annotators | Measures accuracy objectively |
| Pilot | A small batch to calibrate before scale | Exposes guideline gaps cheaply |
| Quality metrics | Agreement, accuracy, boundary tolerance, per class | Defines "acceptable" |
| Audit sampling | Method and rate for buyer and vendor checks | Detects drift over time |
| Rework | Who fixes failures and at whose cost | Avoids disputes |
| Data handling | Environment, access, retention, subcontractors | Protects confidentiality |
| Ownership | Labels, guidelines, derived tools | Clarity on reuse |
| Delivery | Format, versioning, metadata | Usable output |
Pick targets for the metrics based on your task; no universal threshold applies. A boundary tolerance suitable for coarse step segmentation would be inadequate for contact-event labeling.
Quality metrics worth contracting
- Accuracy against gold: per class, not only overall, so rare classes cannot hide behind common ones.
- Inter-annotator agreement: measured on overlap items, with the measure named in the contract (for example, Cohen's kappa for categorical labels or temporal intersection over union for segments).
- Boundary error: for temporal labels, the allowed offset in frames or seconds; for spatial ones, mask or box overlap.
- Completeness: share of required items labeled, and treatment of "uncertain".
- Consistency over time: performance on gold items injected throughout the project, not just at the start.
- Escalation handling: how ambiguous cases are flagged and resolved.
Define how each metric is computed, on what sample, and who computes it. For examples in a particular domain, see step segmentation annotation and defect taxonomies and label schemas.
Acceptance procedure
- Pilot acceptance: the vendor labels a sample; the buyer reviews against gold; guidelines are revised and re-versioned.
- Batch acceptance: each delivered batch is sampled at an agreed rate. A batch fails if any pre-defined metric is below threshold.
- Remediation: failed batches are corrected and resampled; the contract states a cap on rounds before escalation.
- Final acceptance: a full-dataset report of metrics, per class and per batch, is delivered with the labels.
- Hold-back: a share of payment is conditioned on final acceptance.
Run the buyer-side checks inside an acceptance process for the whole delivery; see acceptance testing for deliveries.
Documentation that travels with the labels
Ask for a dataset document so later users understand how labels were produced. Datasheets for Datasets proposes that every dataset be accompanied by documentation of motivation, composition, collection process and recommended uses (Gebru et al., 2018). Data Cards are structured summaries of the essential facts about a dataset that stakeholders need across its lifecycle (Pushkarna et al., 2022). For annotation, include guideline versions, annotator training, quality results and known weaknesses. See datasheets for robotics datasets.
Illustrative acceptance schedule
# Illustrative only - synthetic values; thresholds are set by the buyer
annotation_sow: step_and_hoi_labels_v0
scope: 5,000 clips (synthetic), 4 label types
schema_version: 1.2 # frozen at pilot acceptance
guidelines_version: 1.2
gold_set: 300 items, expert-labeled, hidden from annotators
overlap: 10% of items double-labeled
metrics:
per_class_accuracy_vs_gold: threshold = buyer_defined
temporal_boundary_tolerance: buyer_defined, in frames
agreement_measure: named in contract
audit_sampling: 5% per batch, random, stratified by site
batch_fail_rule: any metric below threshold on any priority class
remediation: vendor corrects at own cost; max 2 rounds
data_handling: buyer-controlled environment; no copies; subcontractors approved in writing
ownership: labels assigned to buyer; vendor retains no copy
Questions to ask vendors
- Who will annotate, how are they trained, and are subcontractors used?
- How do you build and protect a gold set, and how do you measure agreement?
- Can you show past quality reports with per-class results?
- How do you handle ambiguous or occluded cases?
- What tools do you use, and can labels be exported in open formats?
- Where is the data stored, who can access it, and how is it deleted?
- Does the contract assign the labels to the buyer, and what do you retain?
- Can you handle our confidentiality constraints, for example restricted-environment work?
- What is the process and cost when a batch fails acceptance?
Discuss your requirements
If your data needs annotation, describe the label types and quality expectations to SourceX as part of the request. SourceX's role is sourcing, rights review, licensing and delivery coordination; annotation can be arranged case by case, and a request does not guarantee a match. SourceX reviews the requirement and follows up about potential sources. See the humanoid training data guide, the procurement hub, requirements specification template, and the catalog entries for expert annotations and labels and the training data due diligence checklist.
Discuss your humanoid training-data requirements with SourceX
Sources
- Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset summaries across the lifecycle.
- Datasheets for Datasets — documentation of motivation, composition and collection process.