Relabeling Legacy Defect Archives

What is wrong with legacy labels

An archive from years of production has the images and a label per image or per part. The label may be an operator's reject code, a machine-vision rule output, a rework ticket or a free-text note. Over the years, codes were renamed, standards were revised, cameras were replaced and reasons for rejecting drifted.

The result is a set that looks complete but has mixed definitions. Typical problems: the same defect under three codes; one code covering several defects; labels for the part but not for the image shown; false rejects recorded as defects; defects with no location; and good parts that were never confirmed as good.

Relabeling is the repair. Whether to do it, how much, and for which images are procurement decisions that should come before licensing the whole archive. The schema to relabel into is described in defect taxonomy and label schema, and the sibling issue of rater consistency in inter-inspector agreement and label noise.

Decision: how much detail to buy

Published work gives a useful budgeting signal. The mixed-supervision paper introduces KolektorSDD2, with over 3,000 images and several defect types, and reports that mixing a few pixel-labeled samples with image-level labels performs comparably to fully supervised training at much lower annotation cost (Božič et al., 2021). The study does not say how many pixel labels your task needs, and it was run on specific surface-defect sets.

A plan worth considering mixes levels: image-level labels everywhere, class labels where taxonomies are clean and masks on a stratified subset chosen by the pilot. Mask formats and interchange are in defect annotation formats and pixel-level defect segmentation.

A relabeling workflow

1. Triage by sampling. Draw a stratified sample by year, line, camera and original code. Have an expert tag each as correct, wrong, ambiguous or unusable (blur, no part, cropped). The error rates tell you whether relabeling beats starting fresh.
2. Define the target. Fix the label schema, severity scale and the standard revision. Add example images for each class and border cases.
3. Pilot. Relabel a few hundred images with two or more raters. Measure agreement and time per image, and revise the guidelines.
4. Gold set. Build an expert-adjudicated subset used to test annotators before and during production.
5. Production relabel. Assign images in batches; route low-confidence and disagreements to senior review.
6. Audit. Sample accepted batches, track error by annotator and class, and re-open batches that fail.
7. Document. Record which labels are original, relabeled or adjudicated, with versions. Dataset documentation templates such as Data Cards give structure for this (Pushkarna et al., 2022).

Keep the original label next to the new one. Disagreement between them is informative, and a buyer may want to audit both.

Illustrative relabel batch record

# Illustrative only: synthetic fields for tracking relabeling
image_id: img_2271904
source_label: {code: "SRF-3", origin: "operator_reject", year: 2021}
target_label: {class: "scratch", severity: "minor", standard_rev: "C"}
mask_file: null            # only on stratified subset
raters: [ann_04, ann_11]
agreement: "split"
adjudicated_by: "senior_reviewer_02"
status: "accepted"         # accepted | rework | unusable
notes: "glare at top edge; defect partly out of frame"

Roles and who supplies what

The manufacturer or supplier owns the images and knows the standards; an annotation provider supplies trained labelers and tooling; the buyer owns the target schema and acceptance. Domain expertise matters more than headcount: general crowd labelers misjudge subtle finishing defects or weld classes. Plan for subject-matter reviewers from the supplier's quality team, or independent experts, to anchor the gold set.

Rights need an explicit answer. Annotation providers can be engaged as work-for-hire for the labels, but their work does not create rights in the underlying images; the supplier must be able to license those. Customer part imagery needs confidentiality review before outside annotators see it; see confidentiality in defect data.

What relabeling cannot fix

• Missing images. If defects were never photographed, or only the rejected part's final image was kept, no relabel recovers the missing views.
• Poor imaging. Glare, motion blur and low resolution limit what a rater can mark.
• Lost context. Without the original standard, raters apply their own judgment, which adds a new definition to the archive.
• Selection bias. Archives hold what was kept. Relabeling does not add rare defects; see rare defects and class imbalance.
• Leakage. Near-duplicate images across splits stay duplicates after relabeling; see leakage in defect datasets.
• Robot capabilities. Cleaner defect labels do not turn an image archive into manipulation data.

Limitations and additional preparation

Relabeling is serial work that scales with image count, expert time and review depth, so scope by pilot rather than by archive size. Archives from in-line machine vision systems tend to have consistent imaging but labels that mirror the vision rule; manual-inspection photos have more varied imaging and human-written labels. Both need different triage.

Sizing the effort

Relabel cost depends on three things: the number of images touched, the label level and the expert time per image. The pilot gives you the last number. Multiply it by the planned volume at each level, add review time for disagreements and audit sampling, and compare with the cost of collecting fresh labeled images. Sometimes relabeling is cheaper; sometimes the archive is so poorly imaged that new capture is the better route.

Be selective. Relabel first the images that matter most to your claims: borderline items, rare classes and the evaluation slice. Archive images that are clearly good and clearly typical can often keep their image-level label after a spot check. Use the audit to tell which batches can be trusted.

Finally, decide what the delivered package should include. Severity and disposition fields follow a different logic to type labels; see defect severity and disposition labels for how to specify them so they survive a later change of standard.

Questions to ask suppliers and annotation providers

• Can you provide a stratified sample of images with their original labels for a pilot?
• How have defect codes and standards changed across the archive, and is there a mapping?
• What share of original labels came from humans, machine rules or downstream tests?
• Are raw images retained at full resolution, or only compressed copies?
• Who will relabel, with what domain training, and who holds the gold set?
• How will originals and relabels be versioned and delivered?
• How are disagreements adjudicated and audited?
• Who confirms the supplier may license the images to outside annotators?

Discuss your requirements

If you hold a licensed archive or are evaluating one, describe the label schema, standard and permitted uses to SourceX. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation, scoped case by case. SourceX reviews the requirement and follows up about potential sources, and a request does not guarantee a match. For the wider map, see the humanoid training data guide and the inspection, defect and failure data hub. The catalog page on expert annotations and labels describes the label type.

Explore sourcing and licensing options with SourceX

Sources

• Mixed supervision for surface-defect detection: from weakly to fully supervised learning — KolektorSDD2 and mixed supervision cost findings.
• Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset documentation.