Short answer
Leakage in defect datasets occurs when training and test images share something the model can exploit besides the defect: the same physical part, the same production lot, the same line or camera, or near-identical frames. Scores then measure memory, not detection. Specify the unit of separation (part, lot, line, time), require a split manifest, and run duplicate and near-duplicate checks before you accept any evaluation result.
Why inspection data leaks more than most
In a typical image-classification benchmark, two images of different cats are independent. In inspection they often are not. A production line makes thousands of nearly identical parts, a camera captures each part several times, and a defect is shared by every image of the part that carries it. A random image-level split puts some of those images in training and others in test, and a model can score well by recognizing the part, not the defect.
Inspection data also carries quiet correlates of the label. Defective parts may be photographed in a different session, with a different fixture, or by a different shift than good ones. The model learns session, not defect. The result is an inflated score that collapses in production. The problem here is specific to inspection; for the general concept of held-out sets in robotics, see defect detection evaluation sets.
The leakage routes
| Route | How it leaks | Split rule that blocks it |
|---|---|---|
| Same physical part | Several views or repeated captures of one part land on both sides | Split by part ID, never by image |
| Same lot or batch | Lot-specific texture, color or tooling marks identify the label | Split by lot, with whole lots held out |
| Same line, fixture or camera | Setup artifacts correlate with defect status | Hold out a line or camera for evaluation |
| Same session or day | Lighting drift and dust differ by session | Hold out entire sessions or time windows |
| Duplicate or near-duplicate frames | Video frames, re-exports and crops of one image | Deduplicate before splitting; use similarity search |
| Augmentation before splitting | An augmented copy of a test image sits in training | Split first, augment only the training side |
| Reference or golden images | The "good" reference reappears in training | Keep reference images out of evaluation, or fix them across splits and report it |
| Label-derived shortcuts | Class-specific file names, folders, metadata or border artifacts | Strip identifiers; test a metadata-only baseline |
| Multi-view captures | Views of one part split across train and test | Group all views by part before splitting |
What a careful benchmark design looks like
Published benchmark practice shows the direction. MVTec AD 2 contains eight scenarios with more than 8,000 high-resolution images and includes harder cases, such as small defects in large images, transparent or reflective objects and overlapping bulk goods. It provides a test_private_mixed split captured under varied lighting conditions, and the private test set is scored via an evaluation server (Heckler-Kram et al., 2025). Keeping the test labels private reduces the chance of tuning to the test set, and varying lighting tests whether results hold when conditions shift.
That design is a benchmark's, not a purchase template. Your own evaluation set needs its own separation rules, chosen to match the claim you want to make: "works on new lots", "works on a new line", "works next quarter", or "works on a new part".
Specifying splits in a purchase
State the claim first, then the unit of separation:
- New parts, same line: split by part ID.
- New lots: split by lot or batch, ensuring lots are fully separated.
- New lines or sites: hold out an entire line or site; see domain shift across lines.
- Future performance: split by time and train only on earlier data.
- New defect types: hold out a defect class entirely for an "unseen defect" test.
Require the supplier to deliver a manifest listing each image's part, lot, line, camera, session and split assignment, so the splits can be audited. Imaging conditions should be captured per image; the fields are in imaging conditions metadata.
Illustrative split manifest and leakage checks
# Illustrative only - fields and values are examples, not an available dataset
split_manifest:
claim: "generalizes to unseen lots on line_B"
unit_of_separation: lot_id
train: {lots: [L101, L102, L104], lines: [line_A, line_B]}
validation: {lots: [L105], lines: [line_A]}
test: {lots: [L106, L107], lines: [line_B]}
excluded: [reference_images, calibration_targets]
per_image_fields: [image_id, part_id, lot_id, line_id, camera_id, session_id, split, view_id]
checks:
part_overlap_across_splits: 0 required
lot_overlap_across_splits: 0 required
near_duplicate_pairs_across_splits: report count, threshold set by buyer
metadata_only_baseline: must perform near chance on test
filename_or_folder_label_correlation: must be absent
Checks you can run on delivered data
- Group overlap. Join the manifest to the images and confirm that no part, lot or session appears in more than one split.
- Near-duplicate search. Compute image hashes or embeddings, then list cross-split pairs above a similarity threshold you choose. Inspect examples by eye.
- Metadata-only baseline. Train a simple model on file size, timestamps, camera ID and similar fields. If it predicts the label above chance on the test set, shortcuts exist.
- Background-only check. Mask the part and see whether the model still predicts defects from the surroundings.
- Held-out performance drop. Compare performance on a random split with performance on a lot-based or line-based split. A large gap suggests the random split was leaking.
These checks are inexpensive relative to a data purchase, and they apply to your own archive as well as to a supplier's.
Limitations: what split rules cannot fix
Careful splitting prevents known leaks but cannot prove none remain. Unknown correlates, such as a supplier habit of photographing defects from a certain angle, survive any split. A held-out set also only measures what it contains; if rare defect classes are absent from the data, no split will reveal that the model misses them. See rare defects and class imbalance.
A clean split also does not make the evaluation representative. If all test images come from one line in good lighting, the score tells you about that line. Reference-image practice can add its own leakage when good samples are reused; see golden samples and reference images. For parts imaged from many angles, see multi-view inspection imagery. None of this concerns robot control data, which has its own held-out design rules by site, operator and object.
A short example of how leakage inflates a score
Imagine a line that images each part several times from different angles. A random split puts most views of many test parts into training. The model recognizes the part from its scratches and tooling marks and answers correctly without ever detecting a defect. Re-split by lot, and the score falls to what the model can actually do on new lots. The gap between the two numbers is a direct estimate of how much the original evaluation was leaking, and it costs one extra training run to measure.
Questions to ask suppliers
- What identifiers (part, lot, line, camera, session) are available for every image?
- How many images come from each physical part, and from how many parts in total?
- Were captures from the same part kept together when splits were made?
- How were duplicates and near-duplicates detected and removed?
- Was any augmentation applied before splitting, and how can I tell?
- Are good and defective parts captured in the same sessions and with the same fixtures?
- Can you provide a split manifest and the rule behind it?
- Are reference or golden images included in any split?
- Has a metadata-only baseline been run, and what did it show?
- Can I hold back part of the data for a private evaluation I run myself?
Discuss your requirements
If you can state the claim your evaluation must support and the separations it needs, tell SourceX what your evaluation set needs to cover. SourceX reviews the requirement and follows up about potential US business sources of defect imagery with the traceability you need. A request does not guarantee a match.
Start from the humanoid training data guide or the inspection, defect and failure data hub. The catalog pages on evaluation sets and manufacturing quality records are related.
Tell SourceX what your evaluation set needs to cover
Sources
- The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection — benchmark design with varied lighting and a private evaluation-server test set.