Short answer
Multi-site procedure corpora often contain many near-copies: one template, many sites. Specify clustering instead of blind deletion. Group exact and near duplicates into families, keep one canonical text, retain each variant's differences and its license metadata, and split training from evaluation by family. Deduplication improves corpus hygiene; it does not verify accuracy or rights, and it cannot make procedure text teach motion.
Why procedure corpora are unusually repetitive
A company with many sites (say, an illustrative forty) rarely writes forty independent procedures for receiving freight. More often it writes one, and each site edits a copy. Franchise systems can go further: the franchisor issues a manual and units use it with small changes. Contract providers may reuse their own text across clients, and equipment makers' instructions can turn up pasted into customers' documents.
The result is a corpus whose raw size overstates its variety. If you count documents, you may be counting the same procedure many times. If you train and evaluate on random splits, copies of a document can sit on both sides, and the evaluation then measures memorization. The ownership side of the problem is in franchise operations manuals.
At the same time, the differences between near-copies carry information: a site that added a step probably had a reason. Deletion throws that away. A better specification is usually clustering with retained variants.
What "duplicate" can mean
| Level | Definition | Typical method | Buyer decision |
|---|---|---|---|
| Exact | Identical after normalizing whitespace and formatting | Hashing of normalized text | Collapse, keep source list |
| Near-duplicate | Same text with small edits | Commonly shingling and set-overlap estimates, or fuzzy hashing | Cluster; keep diffs |
| Structural | Same step sequence, different wording | Step-level alignment | Cluster for splits; keep wording variety |
| Semantic | Same task, written independently | Embedding similarity | Do not collapse; treat as variants of one task |
| Translation | Same text in another language | Language ID plus alignment | Link as a family |
The thresholds that separate these levels are corpus-dependent. Any threshold you see in a supplier's description should be a parameter they report and justify on your data, not a universal number. Ask them to show borderline pairs.
What to specify in a deduplication job
- Unit. Whole documents, sections or individual steps. Step-level units find shared boilerplate; document-level units find template families. Specify both if you will split by family and also measure step overlap.
- Normalization. What is stripped before comparison: headers, footers, page numbers, revision blocks, site names, dates. Overly aggressive normalization can merge distinct procedures.
- Output. A cluster identifier per document, a canonical member, and a per-member difference summary rather than a reduced file set.
- Provenance carried through. Source, supplier, license terms and permitted uses must stay attached to every member, not just the canonical one.
- Audit. A reviewed sample of clusters, with the false-merge and missed-merge cases reported.
Why license metadata must survive
Merging sources is a common point at which licensing information can be lost. An audit of more than 1,800 text datasets traced their provenance and found that, after re-annotation, 66 percent of analyzed Hugging Face licenses fell in a different use category from the one first listed, often labeled more permissive than the author's own license; the work also reduced unspecified licenses from over 72 percent to 30 percent (Longpre et al., 2023). The datasets in that audit were public text collections, not corporate SOPs, so the numbers are not a measure of your corpus. The lesson transfers: if deduplication collapses members into one canonical text without recording which source contributed what and under what terms, you cannot later answer a rights question.
If a family contains members from three suppliers, the canonical text is only usable to the extent all three licensed it. Record that at the family level, and let the strictest term govern until a contract says otherwise. See who owns a procedure and the data due-diligence checklist.
Illustrative cluster manifest
The manifest is Illustrative: identifiers and values are synthetic.
{
"cluster_id": "fam_0412",
"level": "near_duplicate",
"canonical_doc": "doc_site_003_0077",
"members": [
{"doc": "doc_site_003_0077", "supplier": "supplier_A", "license_ref": "lic_A_01", "diff_from_canonical": []},
{"doc": "doc_site_011_0031", "supplier": "supplier_A", "license_ref": "lic_A_01", "diff_from_canonical": ["step 6 added", "tool name changed"]},
{"doc": "doc_site_058_0019", "supplier": "supplier_B", "license_ref": "lic_B_02", "diff_from_canonical": ["step 3 reordered"]}
],
"strictest_license_ref": "lic_B_02",
"review_status": "sampled_ok"
}
Splits and evaluation
Evaluation sets must be held out by cluster, never by document. If two members of a family land on opposite sides of a split, a model sees the answer in training. Whole families go to one side, and for claims about generalization, hold out by site or supplier as well. Training and evaluation data are different things, and evaluation sets are not to be mixed back into training; procedural reasoning evaluation sets covers construction in more detail.
Clustering also helps measure diversity honestly. Report the number of families next to the number of documents, and use procedure coverage mapping to see which tasks the families cover. For generated or paraphrased procedures that may inflate duplicates, see real vs generated procedures.
What deduplication cannot do
Deduplication is a hygiene step. It cannot confirm that a procedure is correct, current or followed; procedure quality scoring addresses quality separately. It cannot clear rights, since removing a copy leaves the question of the original open. And it cannot change what procedure text is: no motion, force or perception data. A humanoid's control policy still needs robot or teleoperation data, and a cleaned procedure corpus cannot substitute for it.
There is also a risk in over-collapsing. If dedup treats a safety-relevant extra step as noise, the cluster canonical may omit it. Always preserve the differences, and review clusters where members differ in warnings, limits or sequence.
Questions to ask suppliers
- What unit of comparison do you use: document, section or step?
- What normalization precedes comparison, and can you show a before and after example?
- Do you remove duplicates or cluster them, and are removed members recoverable?
- How do you report borderline pairs, and who reviews them?
- Does provenance and license metadata stay attached to every cluster member?
- Can you give the number of families alongside the number of documents?
- How do you treat translations and re-typed copies of the same text?
- Can splits by family be supplied, with documentation of how they were made?
For preparation that reaches beyond deduplication, you can describe your corpus requirements and raise the rest as part of scoping.
Discuss your requirements
If you are specifying a procedure corpus, say how many suppliers, sites and templates you expect and how you plan to split it. SourceX reviews the requirement and follows up about potential sources; a request does not guarantee a match, and preparation work is scoped case by case. See the procedures hub and the humanoid training-data section, plus the SOP corpus evaluation checklist for the broader review.
Discuss your humanoid training-data requirements with SourceX.
Sources
- The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI — provenance and license mislabeling in public text datasets.