Short answer
A defect taxonomy defines the classes annotators assign; a label schema defines every field recorded with them, such as location, severity and disposition. Design both before collecting images, because labels cannot be recovered later. Tie each class to a written acceptance definition, keep the hierarchy shallow enough to label consistently, and test it on a pilot set before scaling.
Why label design decides dataset value
Two datasets can contain identical photographs and have very different worth. One labels "defect: yes/no". Another labels class, location, size, severity and what the factory did with the part. Only the second can train a classifier, support disposition decisions or evaluate a model against a customer specification.
Most defect datasets are bought for the images, then relabeled at cost when the taxonomy does not match the buyer's. Writing the taxonomy first reduces that rework and makes supplier archives easier to assess, because you can ask how their existing labels map to yours. If a supplier already holds labeled archives, see relabeling legacy defect archives.
What the schema can support - and what it cannot
A well-designed schema supports training classifiers and segmenters, measuring inter-annotator agreement, defining pass/fail evaluation and mapping defect types to dispositions. It does not create images of rare defects, make an inconsistent inspector consistent, or settle what a customer considers acceptable. It also says nothing about how a defect behaves when a robot handles the part; that needs handling and manipulation data.
Labels also inherit the source's limits. A taxonomy written around visible surface defects will not capture internal flaws that need X-ray or functional test results, and those require different imaging. See X-ray and CT inspection images.
Principles for a usable taxonomy
Define classes by observable appearance and acceptance, not by cause. "Scratch" and "porosity" can be labeled from an image. "Tool wear" is a cause an inspector infers. Record causes in a separate field if known.
Keep the hierarchy shallow. Two or three levels (family, type, optional subtype) are usually labelable consistently. Deeper trees multiply disagreement, and the leaf classes become too rare to learn from.
Make classes mutually exclusive, or say how to combine them. If a part can have a scratch and a dent, decide whether that is two labels or one. Decide how overlapping regions are handled.
Include non-defect classes. Cosmetic variation, dust, marks that wipe off, and acceptable texture are not defects but cause false alarms. Naming them lets a model learn the difference.
Add an explicit "uncertain" label. Borderline samples are real; forcing a choice produces noise. See borderline samples.
Link each class to its standard. Where a published or customer standard defines acceptance, cite the clause. IPC-A-610 is a well-known example in electronics: it defines visual acceptance criteria and three product classes, Class 1 general, Class 2 dedicated service and Class 3 high performance, the strictest (IPC, 2024). The same defect can be acceptable under one class and rejectable under another, so the class of product must be a recorded field. Check the license for standards text before quoting it in guidelines. See acceptance standards as label definitions.
How published benchmarks structure labels
MVTec AD spans 15 categories with more than 70 defect types and pixel-precise ground truth for all anomalies (Bergmann et al., 2021). Its defect types are specific to each object category, which is a pattern worth copying: define defect types per product family, then group them into shared families (surface, shape, missing material, contamination) only if a model needs cross-product classes. A flat shared list across unrelated products tends to have vague classes.
Illustrative label schema
# Illustrative only - synthetic classes and values
schema_version: 0.3
product_family: housing_A
product_class: "customer-spec-grade-2" # which acceptance standard applies
classes:
surface:
- scratch
- scuff_cosmetic # non-defect
- contamination
shape:
- dent
- burr
material:
- void_or_porosity
- missing_feature
other:
- uncertain
- not_a_defect
fields_per_label:
defect_class: one_of(classes)
geometry: bbox | polygon | mask # required for priority classes
location_zone: zone_01..zone_08 # from drawing
size_mm: measured or estimated flag
severity: minor | major | critical
disposition: accept | rework | scrap | hold
standard_ref: clause_id or null
annotator_id: pseudonymous
confidence: high | medium | low
image_fields: station_id, camera_id, lighting_id, timestamp, part_serial_hash
Parameters the buyer sets
- Which classes need masks and which need only boxes or image-level tags.
- Size thresholds, for example the smallest scratch that counts, defined against image resolution.
- Whether severity and disposition (see defect severity and disposition) come from the factory's records or from new annotation.
- How many annotators label each image and how conflicts are adjudicated.
- Whether labels are final or versioned as the taxonomy changes.
Acceptance tests for a labeled set
- Schema validation: every label uses a valid class and required fields; no free-text classes.
- Pilot agreement: two or more annotators label the same sample; report agreement per class and review classes with low agreement. See inter-inspector agreement.
- Gold-set audit: a subset labeled by senior inspectors is compared against vendor labels.
- Coverage report: counts per class, per product family, per site and per imaging condition.
- Leakage check: near-duplicate images do not span training and evaluation splits.
- Traceability: each label maps to an annotator, guideline version and, where relevant, standard clause.
For contracting annotation work against these tests, see annotation vendor requirements and acceptance criteria. For file formats, see defect annotation formats.
Questions to ask suppliers
- Does the supplier already use a written defect taxonomy, and can you see it?
- How were existing labels produced: factory inspectors, quality engineers or outside annotators?
- Do labels reflect a customer or industry standard, and which version?
- Is severity or disposition recorded, and by whom?
- How are borderline and non-defect cases labeled?
- Can classes be mapped to your taxonomy without re-inspecting images?
- Are there guideline documents, and are they included in the license?
- Were labels revised over time, and can the history be provided?
Discuss your requirements
If you already have a draft taxonomy, share it with SourceX as part of your inspection data request. SourceX investigates whether US businesses hold images and labels that fit, checks the data and the supplier's licensing permissions, and follows up about potential sources; annotation and specialist capture partners can be arranged case by case, and a request does not guarantee a match. Start from the humanoid training data guide or the inspection and defect data hub, and see the catalog entry for expert annotations and labels and manufacturing quality records. Failure classes for robot tasks are covered separately in failure taxonomies for embodied tasks.
Discuss your humanoid training-data requirements with SourceX
Sources
- The MVTec Anomaly Detection Dataset — defect types per category with pixel-precise ground truth.
- IPC-A-610 Acceptability of Electronic Assemblies — visual acceptance criteria and product classes as an example of a defining standard.