Short answer
In most defect archives the classes that matter most have only a handful of examples, often from a few physical parts. Rarity limits what you can train and, more sharply, what you can measure. Count distinct defective parts per class rather than images, match the method to those counts (anomaly detection, few-label supervision or synthetic defects), and reserve real defects for evaluation before spending them on training.
Why rarity, not volume, is the binding constraint
A controlled process produces good parts by the million and defects slowly. The defects that matter most, such as cracks, missing features and contamination in a sealed area, are often the rarest because the process has been engineered against them. An archive can hold terabytes and still contain three examples of the one failure your humanoid must never pass.
Public benchmarks show the same shape. VisA holds 9,621 normal images against 1,200 anomalous ones across 12 objects (Zou et al., 2022). MVTec AD trains on defect-free images only and keeps every defect for testing (Bergmann et al., 2021). Real archives are usually more skewed than either, because benchmark curators select defects to fill classes.
General sourcing of defect imagery is covered in the guide to sourcing defect image datasets. This page is about the planning problem that rarity creates.
Count the right unit
Suppliers report image counts because that is what their storage reports. For rare classes the useful unit is the distinct defective part, and often the distinct lot.
| Unit | What inflates it | Why it matters |
|---|---|---|
| Images | Multiple cameras, re-inspection, burst capture, re-runs after a recipe change | Ten images of one cracked part teach one crack |
| Defect instances | Several defects on one part counted separately | Useful for localization, misleading for part-level verdicts |
| Distinct defective parts | Nothing, if serials or part IDs are recorded | The realistic sample size for a class |
| Distinct lots or production days | Nothing | Governs whether a model saw one process upset or many |
Illustrative class count request
Ask for counts at every level, per class, before any sample is sent. The synthetic example below shows how a healthy-looking image count can collapse.
| Class (synthetic) | Images | Defect instances | Distinct parts | Distinct lots |
|---|---|---|---|---|
| scratch_cosmetic | 4,120 | 4,870 | 1,310 | 212 |
| dent | 610 | 640 | 188 | 41 |
| crack_edge | 96 | 97 | 11 | 4 |
| missing_thread | 38 | 38 | 6 | 2 |
| contamination_seal | 22 | 25 | 3 | 1 |
In this illustration, contamination_seal looks like 22 examples and is really one lot. A model trained on it learns that lot.
What published work shows you can do with few defects
Several lines of research address scarce defect data, and each implies a different purchase.
- Learn from a few labeled defects. A two-stage segmentation-and-decision network was trained on KolektorSDD, a 399-image set, and learned from a small number of defective samples (Tabernik et al., 2019). A follow-up mixed a few pixel-labeled samples with weaker image-level labels and performed comparably to full supervision at much lower annotation cost (Božič et al., 2021). The purchase implication: a few precise masks on real defects can be worth more than many loose labels.
- Model normal appearance only. PatchCore fits a memory bank of features from defect-free images and flags anything far from it, reaching image-level AUROC up to 99.6% on MVTec AD (Roth et al., 2021). Benchmark scores do not transfer to your line, but the method changes what you buy: many nominal images across lots, lines and lighting, plus a small labeled defect set for testing.
- Generate defect-like training signal. CutPaste learns a representation by classifying normal images against copies with a patch cut and pasted elsewhere (Li et al., 2021). The DAGM 2007 benchmark used artificially generated but realistic defects on textured surfaces (Wieler et al., 2007). Synthetic defects help when the defect is visually simple; when and how they mislead is the subject of synthetic vs real defects.
- Accept that "normal" is contaminated. An archive's pass images include escapes the old system missed. Real-IAD defines a fully unsupervised setting in which defective images are present in the training pool (Wang et al., 2024), which is closer to a plant archive than a curated defect-free set.
The choice between modeling normal appearance and learning named classes is covered in anomaly detection vs supervised defect data.
Matching approach to counts
The thresholds that separate these rows are yours to set; they depend on the model, the defect's visual complexity and the claim you need to make. There is no universal minimum per class.
| Situation for a class | Approach that fits | What to buy | Main risk |
|---|---|---|---|
| Defect type unknown in advance, or almost never seen | Anomaly detection on nominal images | Nominal images across lots, lines and lighting; a held-out defect set for testing | Flags anything unusual, but cannot name the defect |
| A few distinct defective parts | Few-label or mixed supervision | Pixel masks on the few real defects; image-level labels elsewhere | Overfitting to the specific parts and lot |
| Defect can be reproduced on real parts | Commissioned imaging of physical samples | Access to retained rejects or seeded defects | Staged defects can differ from production ones |
| Defect is visually simple (texture, spot, streak) | Synthetic or augmented defects | Nominal images plus a real test set | The model learns synthetic artifacts |
| Many distinct parts across lots | Standard supervised training | Per-class counts by split, with severity | Label noise dominates once counts are high |
Evaluation is where rarity hurts most
Rare classes are hard to train and harder to test. With 10 defective parts in a test class, each miss moves measured recall by 10 percentage points; with 3, a single miss moves it by 33. A handful of examples cannot support a claim about a low escape rate, however good the training method.
That leads to three rules for the purchase:
- Reserve first, train second. Decide which real defects go to the test set before anything is used for training. If a class has too few distinct parts to fill both, keep the real ones for testing and train with nominal data, synthetic defects or both.
- Split by part and lot, not by image. Images of the same physical defect must never sit on both sides of the split. See defect detection evaluation sets for a held-out design.
- Report per class at an operating point. Pooled accuracy is dominated by nominal images. Escapes and false rejects carry different costs; the DAGM 2007 competition, for example, weighted false positive and false negative decisions asymmetrically at 1:20 (Wieler et al., 2007). Set your own cost ratio from the line's consequences.
Ways to grow a rare class, and what each costs
- Pool across lines, plants or years. More distinct parts, at the price of camera, lighting and process differences; see domain shift across lines.
- Image retained physical rejects. Quarantine areas and material-review cages may hold parts that never reached a camera archive. Re-imaging them on a controlled rig is often the fastest route to more real examples; physical defect samples covers how to source and image them.
- Collect near-threshold parts. Parts inspectors disagree on sharpen a model's decision boundary; see borderline samples.
- Widen the retention window. If the supplier's system overwrites images after weeks, ask whether rejects can be retained going forward under the license.
- Generate defects. Cheapest per image and least trustworthy for evaluation.
The same rarity problem appears in task video, where failures and recoveries are scarce; rare events in demonstrations covers that side.
What this data cannot support
Rare-defect images, however well counted, train recognition. They do not show how a humanoid should pick up, turn or set aside a cracked part, and they are captured from fixed inspection cameras rather than a robot's head or wrist view. Some rare defects are only visible at a particular angle or under raking light, which a fixed archive may never have recorded. Plan robot-view images and handling data separately; active inspection by humanoids explains why viewpoint and handling matter.
Questions to ask suppliers
- For each defect class, how many images, defect instances, distinct parts and distinct lots do you hold?
- Are part serials or IDs recorded per image, so duplicates of one part can be grouped?
- Which classes come from a single process upset, supplier lot or tool failure?
- Were rare defects confirmed by a final disposition, or only flagged by the camera or an inspector?
- Do you keep nominal images, and over what span of lots, lines and dates?
- Are physical rejects retained, and could they be re-imaged under controlled conditions?
- How long does your system keep images, and can rejects be retained longer from now on?
- Has the class definition changed during the archive period?
Discuss your requirements
If rare defect classes decide whether your inspection model can be trusted, tell SourceX which classes, parts and lines your evaluation needs. SourceX looks for US businesses that hold suitable inspection data, checks the data and the supplier's licensing permissions, and coordinates pricing, allowed uses and delivery in a license. It reviews each request and follows up about potential sources; a request does not guarantee a match. The inspection, defect and failure hub and the humanoid training data guide cover neighboring data types, and the catalog entry for manufacturing quality and inspection records describes the record type.
Tell SourceX what your evaluation set needs to cover
Sources
- SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation — VisA normal and anomalous image counts.
- The MVTec Anomaly Detection Dataset — defect-free training with defects reserved for testing.
- Segmentation-Based Deep-Learning Approach for Surface-Defect Detection — learning from few defective samples on a small dataset.
- Mixed supervision for surface-defect detection: from weakly to fully supervised learning — few pixel labels mixed with weaker labels.
- Towards Total Recall in Industrial Anomaly Detection — anomaly detection fitted on nominal images only.
- CutPaste: Self-Supervised Learning for Anomaly Detection and Localization — synthetic anomalies by cut-and-paste augmentation.
- Weakly Supervised Learning for Industrial Optical Inspection (DAGM 2007) — artificially generated defects and asymmetric error costs.
- Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection — defective images present in the training pool.