Inter-Inspector Agreement and Label Noise in Defect Data

Why disagreement matters to a buyer

Production labels are rarely produced for machine learning. They are the outcome of a decision: an operator scrapped a part, a quality technician wrote a code, a machine-vision rule fired. Each of those decisions reflects shift pressure, fatigue, the inspector's training, the standard in force that month and the cost of a wrong call.

Two inspectors looking at the same scratch can reasonably disagree on whether it is cosmetic, whether it exceeds the limit and whether it needs rework. When a model is scored against one of them, its measured error includes the other's disagreement. This is the ceiling effect: if two experienced humans agree on only part of the borderline cases, a model that matches either is already near the limit of what the labels can show.

This page is a checklist for evaluating an offered dataset. If you are designing new labels instead, start with defect taxonomy and label schema. If the archive exists but its labels are unreliable, see relabeling legacy defect archives.

What agreement data can and cannot tell you

Agreement statistics describe consistency, not correctness. Two inspectors can agree and both be wrong against the written standard, and one can be right where the other is not. Agreement also depends on the sample: a set made mostly of clear good parts and clear scrap shows high agreement and says little about the boundary.

Agreement data does not show whether labels match your acceptance criteria. A supplier's scrap decision may reflect a customer-specific limit that does not transfer. It also does not substitute for an independent held-out set; see defect detection evaluation sets.

Where label noise comes from

• Boundary ambiguity. The same flaw sits on either side of a threshold depending on who looks. This is where borderline samples concentrate.
• Vague definitions. "Surface damage" can cover a scratch, a dent, a stain and a mold mark. See acceptance standards as label definitions.
• Process-driven labels. A part labeled good because it passed a downstream test, or bad because a lot was held for an unrelated reason.
• Rule-driven labels. A machine-vision system flagged an image and nobody checked it, so the label records the rule rather than the part.
• Weak labels. Image-level tags used where pixel-level masks would be needed. Published work on KolektorSDD2, a set with over 3,000 images, reports that mixed supervision with a few pixel-labeled samples can perform comparably to full supervision at much lower annotation cost (Božič et al., 2021). The result is useful for budgeting, but it assumes the few detailed labels are correct.
• Drift. Standards, tooling and personnel change across years of an archive.

Illustrative checklist

Criteria are examples; the buyer sets the thresholds.

Data Cards were proposed as structured summaries of the facts stakeholders need across a dataset's lifecycle (Pushkarna et al., 2022); asking a supplier to fill in the label-process fields is a reasonable, low-cost request.

Illustrative agreement report (synthetic values)

# Illustrative only: synthetic structure for a supplier agreement report
dataset: lineB_housing_2025
labeled_images: 1000
repeat_labeled_subset: 200
raters: [inspector_A12, inspector_B07, inspector_C03]
agreement_by_class:
  scratch: {kappa: 0.xx, n: 60}
  dent: {kappa: 0.xx, n: 35}
  contamination: {kappa: 0.xx, n: 45}
agreement_by_severity: {minor: 0.xx, major: 0.xx, critical: 0.xx}
adjudication: "senior quality engineer settles splits; recorded"
standard_version: "customer spec rev C, 2025-03"

How to use the numbers

Report model accuracy next to the human agreement rate on the same subset. A model within the human agreement band is not necessarily better or worse than a person. A model far above it usually signals leakage or an evaluation set that is too easy.

Split reporting by class and severity. Aggregate accuracy hides the fact that most of the disagreement sits in a few minor-defect classes. Where labels will be used for an evaluation set rather than training, require an adjudicated subset held out by lot or line, and keep it out of training.

Limitations and additional preparation

Repeat labeling costs inspector time, and a supplier may not have done it. If it did not, the buyer can commission a sample relabel on a stratified subset, which gives an agreement estimate without relabeling everything. Relabeling images of customer parts raises its own confidentiality questions; see confidentiality in defect data.

Agreement is also not a property of the images alone. It changes with the label scheme, the training of the raters, the display used and the instructions. Re-measure when any of those change. Nothing here applies to robot manipulation data, which has different failure modes and different label problems.

Applying the checklist to a real offer

Treat the checklist as a sequence rather than a scorecard. Start with documentation: if the supplier cannot describe how labels were produced, the other checks have nothing to stand on. Next ask for the repeat-labeled subset and compute agreement yourself rather than accepting a summary figure. Then look at where the disagreement sits. If it clusters in one or two classes, you may be able to merge those classes or drop them from the first release. If it spreads across everything, the labeling process, not a few classes, needs work.

Be explicit about what you will do with noisy labels. Options include discarding items where raters split, training with soft labels that reflect rater votes, using noise-robust losses, or relabeling a priority slice. Each option has a data requirement: soft labels need multiple votes per item, and discarding needs enough items to spare. Decide which you plan to use before you choose what to buy, because it determines whether you need several independent labels per image.

For evaluation sets the bar is higher. Use the strictest labels you can afford, with adjudication, and record which items were contested so the report can show performance on the clear and contested subsets separately.

Questions to ask suppliers

• Were any images labeled by more than one person, and what share?
• How did you settle disagreements, and who had the final say?
• Is the label the inspector's call, a machine-vision rule or a downstream test result?
• What defect definitions and standard version applied, and did they change over the archive?
• Can you split agreement by class and by severity, not only overall?
• Were the same inspectors used across lots and lines?
• Is there a clean subset you would stand behind, and how was it chosen?
• What was never labeled, such as images discarded before disposition?

Discuss your requirements

If label quality will decide whether a defect archive is usable, tell SourceX what your evaluation set needs to cover, including the agreement evidence you would require. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match. For the wider map, start at the humanoid training data guide or the inspection, defect and failure data hub. The catalog page on manufacturing quality records describes a related record type.

Tell SourceX what your evaluation set needs to cover

Sources

• Mixed supervision for surface-defect detection: from weakly to fully supervised learning — mixed supervision with few pixel labels; KolektorSDD2 size.
• Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI — structured dataset documentation for stakeholders.