Short answer
Inspection models trained on one line often lose accuracy on another because lighting, cameras, fixtures, material batches, part variants and defect mixes all change. Public benchmarks now include varied lighting and multi-view capture to test this. When buying defect data, specify the variation you need, require line, lot and time metadata, and hold out whole lines or plants for evaluation.
The problem in one scenario
A buyer licenses a year of images from one production line, trains a defect detector and measures high accuracy on a random split. The model goes to a second plant that makes the same part. Accuracy drops. Nothing is wrong with the model; the second plant has a different camera model, an LED panel that ages differently, a darker fixture, a new resin supplier that changes surface gloss, and a defect mix dominated by a flaw the first line rarely saw.
A random split within one line measures whether the model memorized that line. The question the buyer cares about is whether it works on a line it has not seen. This page explains the sources of shift, what benchmarks reveal about it, and what to specify when purchasing. For how images should be documented, see imaging conditions metadata.
Sources of shift
| Source | Examples | What to record |
|---|---|---|
| Optics and lighting | Lens, sensor, exposure, strobe angle, aging lamps | Device model, settings, lamp hours |
| Fixturing | Part pose tolerance, tray design, background | Fixture ID, nominal pose |
| Material and process | Resin lot, coating batch, tool wear | Lot, supplier, process step |
| Product variant | Color, finish, geometry revision | Part number, revision |
| Defect mix | Which flaws occur and how often | Defect code by lot |
| Labeling | Different inspectors and standard versions | Labeler role, standard revision |
| Time | Seasonal humidity, tooling changes | Timestamp, shift |
Most archives carry a few of these fields. Few carry all, and without them a buyer cannot tell which shift explains an error.
What public benchmarks show
Benchmarks have moved toward harder, more varied conditions. MVTec AD 2 includes eight scenarios with more than 8,000 high-resolution images and cases such as small defects in large images, defects at the image border, high normal variability, transparent or reflective objects and overlapping bulk goods. It also provides a test_private_mixed split captured under varied lighting conditions, and its private test set is scored through an evaluation server (Heckler-Kram et al., 2025). The earlier MVTec AD has 5,354 images across 15 categories, with only defect-free images in training (Bergmann et al., 2021).
Real-IAD contributes 150K high-resolution images of 30 objects with multi-view capture, and proposes a fully unsupervised setting in which the training set contains defective images (Wang et al., 2024). That setting resembles real archives better than a perfectly clean nominal set. The design choices in these datasets are useful as a checklist even if you never use the datasets themselves.
The MVTec datasets are released under CC BY-NC-SA 4.0, which restricts commercial use, and a benchmark result does not show how a model behaves on your product. The numbers describe what each study built, not a requirement for yours.
Decision table: what variation to buy
| Your goal | Prioritize | Evaluation design |
|---|---|---|
| One fixed line, stable setup | Deep coverage of that line over time | Hold out later time periods |
| Rollout to several plants | Images from at least two plants or lines | Hold out a whole plant |
| New product variants | Variant coverage and geometry revisions | Hold out a variant |
| Changing illumination | Images at varied lighting settings | Hold out a lighting setting |
| Rare defects | Defect mix across sources | Hold out by lot, not by image |
| Cold start with few defects | Many defect-free images per line | Test on the target line's defects |
Deciding between anomaly methods that need only defect-free images and supervised methods that need labeled defects interacts with this table; see anomaly detection versus supervised defect data. PatchCore-style cold-start methods fit on nominal images alone (Roth et al., 2021), so shift in the nominal appearance matters more than the defect mix.
Illustrative split design
# Illustrative: synthetic IDs and policy for a cross-line evaluation
train:
lines: [plant_A_line_1, plant_A_line_2]
period: 2024-01..2024-12
validation:
lines: [plant_A_line_2]
period: 2025-01..2025-03
test_unseen_line:
lines: [plant_B_line_1] # never used in training or tuning
test_unseen_variant:
part_revision: [rev_D] # appears only here
rules:
- no_lot_shared_across_splits
- no_near_duplicate_shots_across_splits
- report_metrics_per_line_and_per_defect_class
Leakage across lots or near-duplicate shots can hide shift entirely; see leakage in defect datasets and defect detection evaluation sets.
What this means for purchasing
- Buy for the deployment distribution, not the cheapest archive. A large set from one line can be less valuable than a smaller one from several lines.
- Ask for the metadata that explains shift. Without device, lot and line fields, a held-out line cannot be defined. Archives from in-line machine vision systems usually carry more of this than ad-hoc photos.
- Separate evaluation supply. If you will validate on an unseen plant, licensing a plant's images only for testing is a distinct, smaller purchase.
- Plan for adaptation. Even good coverage rarely removes the need to fine-tune on a few images from the target line. Ask whether the supplier can provide a small calibration set from the destination line.
- Consider multi-view capture where defects appear only from some angles; see multi-view inspection imagery.
Limitations
Supplier data from different plants may come from different companies, with different labeling rules, which adds label shift to image shift. Combining sources can also create false confidence if the plants share a supplier or a defect-prone resin. More variation is not always better: contradictory standards across plants can add label noise that outweighs the benefit, as discussed in inter-inspector agreement and label noise.
This page is about image-based inspection. It does not address robot manipulation, where shift involves embodiment, grippers and contact, which image archives cannot capture.
A practical evaluation sequence
Before purchasing, run a small transfer test. Ask for a sample from two lines of the same product, train a simple baseline on one and test on the other. A large drop tells you the supply you need has to include multiple lines; a small drop suggests that deeper coverage of a single line may be enough. The test does not predict final performance, but it prices the risk.
Then plan how labels will be harmonized. If each line used its own defect codes, build the mapping before training, and have a reviewer re-label a sample from the second line against your scheme. Otherwise the model's drop on the second line mixes image shift and label shift, and you cannot tell which to fix.
Last, schedule re-evaluation. Lines change: cameras are replaced, tooling wears and suppliers switch materials. Keep a held-out slice from the most recent period and re-run it periodically, so you notice drift before the line does. A one-off purchase will not stay representative indefinitely.
Questions to ask suppliers
- How many lines, plants and part revisions do the images cover?
- Which camera models, lenses and lighting setups were used, and did they change?
- Can I split by line, lot and time period using fields in the delivery?
- Are defect codes and standards consistent across lines, or mapped?
- What share of images per line are defect-free, and what defect classes appear on which line?
- Were any lines recalibrated or retooled during the period?
- Can you provide a small sample from a second line for a transfer test before a full license?
- Which lines or customers are excluded from licensing for confidentiality reasons?
Discuss your requirements
If cross-line generalization is the risk you are buying against, tell SourceX what your evaluation set needs to cover, including which lines, plants or variants must stay unseen. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates the license and delivery. A request does not guarantee a match. Start with the humanoid training data guide or the inspection, defect and failure data hub. The catalog entry on manufacturing quality records describes related records.
Tell SourceX what your evaluation set needs to cover
Sources
- The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection — harder scenarios, varied-lighting split and evaluation server.
- Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection — multi-view capture and defects in training.
- Towards Total Recall in Industrial Anomaly Detection — cold-start methods fit on nominal images only.
- The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection — earlier benchmark with defect-free training data.