Short answer
A pixel-level defect segmentation specification defines the image set, the class list, how mask boundaries are drawn, how overlapping and borderline defects are handled, and how masks are checked. Masks cost far more than image-level labels, and published work suggests a small pixel-labeled subset can sometimes be enough. Decide per use whether you need masks for training, for evaluation, or only for the evaluation split.
When a mask is worth paying for
A pixel mask answers "where, and how much" rather than "is there a defect." That matters when a downstream decision depends on defect area or position, when the inspection standard sets size limits, or when you must debug why a detector fires. It matters less when the decision is a binary accept or reject.
Masks are the most expensive label type in inspection. Published work gives a useful lever: Božič et al. introduced KolektorSDD2, with over 3,000 images and several defect types, and reported that mixed supervision using only a few pixel-labeled samples performed comparably to full supervision at much lower annotation cost (Božič et al., 2021). Results depend on the dataset and method, so test the trade-off on your own parts before committing to dense masks everywhere. The earlier segmentation-based approach on electrical commutators likewise learned from few defective samples (Tabernik et al., 2019).
This page is the annotation brief. For class design see defect taxonomy and label schema; for sourcing the underlying images see sourcing defect image datasets. Broader context is in the inspection cluster and the humanoid data section.
What masks support and what they do not
Masks support segmentation and localization models, defect-area measurement, evaluation of anomaly maps against ground truth, and error analysis. MVTec AD, for example, provides pixel-precise ground truth for all anomalies across 15 categories with more than 70 defect types (Bergmann et al., 2021). VisA offers image- and pixel-level labels (Zou et al., 2022).
Masks do not give a humanoid any manipulation skill. They describe an image, not a handling action. Whether a robot should pick up, rotate, or re-present a part for a second view is a separate control question, covered in active inspection by humanoids. Masks also inherit every bias of the images: one line, one camera, one lighting setup.
Illustrative annotation specification
The block below is Illustrative, with synthetic values. Adapt the fields to your parts.
# Illustrative mask annotation spec - synthetic values
dataset: line_A_castings_v1
image_unit: single_view_png_16bit
mask_format: png_indexed # one file per image, class index per pixel
classes:
0: background
1: scratch
2: dent
3: porosity
4: contamination
255: ignore # borderline, unreadable or disputed pixels
boundary_rule: "tight mask on visible defect edge; no halo; anti-aliasing off"
min_defect_pixels: 12 # smaller marks go to ignore, not class
overlap_rule: "higher-severity class wins; log overlaps in sidecar"
multi_view: "mask each view separately; link views by part_id"
sidecar_fields: [image_id, part_id, lot_id, line_id, camera_id, annotator_id, review_status, disposition]
qa:
double_label_rate: 0.10
reviewer_sample_rate: 0.20
acceptance_iou_vs_adjudicated: 0.70 # buyer-set example
split_keys: [lot_id, line_id]
Decisions the specification must settle
File format. Indexed PNG, COCO polygons and run-length encodings behave differently for thin defects: polygons smooth them and run-length encodings keep them exact. Compare the options in defect annotation formats and fix one before labeling starts, because converting later can shift boundaries by a pixel or more.
Class granularity. Fewer classes cost less and agree more. Ask which distinctions change a disposition. If scratch and abrasion lead to the same decision, merge them.
Boundary rule. Without a stated rule, annotators draw different edges, and IoU between annotators caps what any model can score. Decide whether the mask covers the visible mark only, the visible mark plus a margin, or the standard-defined zone.
Ignore regions. A dedicated ignore label for borderline or unreadable pixels keeps noise out of both training and scoring. See borderline samples.
Thin and tiny defects. Hairline cracks may be one to two pixels wide. State the minimum width, the resolution at which labeling happens, and whether masks are drawn on the original image or a downscaled copy. MVTec AD 2 deliberately includes small defects in large images and defects at image borders, which stress exactly these choices (Heckler-Kram et al., 2025).
Transparent and reflective parts. Specular highlights can look like defects and real defects can vanish at some angles. Require the imaging conditions to be recorded alongside masks; see imaging conditions and metadata.
Normal-only training images. Anomaly methods train on defect-free images, so masks may only be needed on test images. The trade-offs are in anomaly detection versus supervised defect data.
Quality and acceptance checks
- Double labeling on a sample, scored with IoU or Dice per class, and reported per annotator pair. Low agreement signals an ambiguous rule, not only careless work; see inter-inspector agreement.
- Adjudication by a senior inspector who owns the standard, with the adjudicated mask stored next to the originals.
- Geometric sanity checks: mask size equals image size, no class indices outside the list, no masks on images flagged defect-free.
- Class-wise review of rare classes first, because errors there cost most. See rare defects and class imbalance.
- Disposition link: each masked image should map to the actual accept, rework or scrap outcome where that exists, as discussed in inspection images linked to outcomes.
- Leakage test: confirm that train and evaluation images do not share parts, lots or near-duplicate frames.
Cost drivers and where to save
Cost scales with defects per image, boundary complexity, resolution, and the expertise annotators need. A scratch on a flat surface is quick. Porosity clusters on a casting or solder-mask defects on a dense board are slow and need trained eyes.
Ways to reduce spend without hollowing out the label: mask only the evaluation split densely; use bounding boxes or image-level labels for the training pool; pre-label with a model and have people correct it; and sample the long tail of normal images rather than labeling every one. Each shortcut changes what the label can support, so document it. If the masks will anchor claims about a model, keep a dense, adjudicated, held-out set as described in defect detection evaluation sets, and never let it drift into the training pool.
Sourcing routes
Existing archives sometimes carry bounding boxes or image-level flags from a line's vision system but rarely pixel masks, so masks usually mean relabeling. Legacy images may also lack consistent capture settings; see relabeling legacy defect archives. Annotation can be done by the data holder, by your team, or by a specialist provider with domain staff. Inspectors who know the standard are scarce, and generic crowd annotators tend to mis-draw cosmetic defects.
Public benchmarks are licensed for research. MVTec AD is released under CC BY-NC-SA 4.0, which restricts commercial use, so a commercial buyer should not assume it can be used as training data. This is not legal advice; read the license and ask counsel.
Limitations and extra preparation
Masks describe where a defect is in a particular image, captured with a particular camera and light. They do not transfer cleanly to a different optical setup, and they say nothing about how a robot should handle the part. Boundaries on scratches, stains and low-contrast pits are inherently uncertain, so two careful annotators will disagree, and a model scored against one annotator's masks partly learns that person's style. Rare defect classes may have too few examples to train or to evaluate with confidence; how many are needed depends on the model, the task and the claim you want to make, so no general figure applies.
Extra preparation is usually needed: agreeing the class list and the boundary rules before annotation starts, re-annotating a subset to measure agreement, converting between tool formats, and grouping images by part and lot so the same defect never appears on both sides of a split. If images come from an existing archive, redaction of part numbers or customer marks may crop or blur the very regions you need, which reduces training value.
Questions to ask annotation suppliers
- Who draws the masks, and what standard training do they hold on our defect classes?
- What boundary rule do you apply, and can you show worked examples?
- How do you treat defects smaller than your minimum pixel count?
- What inter-annotator IoU do you report, per class?
- Which tool and format do you export, and can it round-trip with our pipeline?
- How do you handle images where the annotator is unsure?
- Who owns the masks, and can the data holder license them for model training?
Discuss your requirements
If you need defect imagery from US manufacturers and a clear plan for masks, describe the parts, defect classes and label needs to SourceX. SourceX reviews the requirement and follows up about potential sources and rights; annotation can be arranged case by case, and a request does not guarantee a match.
Tell SourceX what your evaluation set needs to cover
Sources
- The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection — Pixel-precise ground truth for all anomalies across 15 categories and 70+ defect types.
- Segmentation-Based Deep-Learning Approach for Surface-Defect Detection — Segmentation-first defect detection that learns from few defective samples (KolektorSDD).
- Mixed supervision for surface-defect detection: from weakly to fully supervised learning — KolektorSDD2 and results showing mixed supervision with few pixel-labeled samples can match full supervision.
- The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection — Harder cases for masks: small defects in large images, border defects, transparent and reflective objects.
- SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation — VisA dataset with image- and pixel-level labels.