Short answer
A defect detection evaluation set is built from real parts held out by lot, line, camera or time, labeled by a stricter process than production labels, and sized by distinct defective parts per class rather than image count. It includes nominal and borderline parts, is never used for tuning, and is scored per class at a fixed operating point. Use the checklist below to accept or reject an offer.
Start from the claim
An evaluation set proves something specific, and the claim decides what must be held out. "Catches crack_edge on line 3" needs different parts from "works on next quarter's production" or "works on a new part number". Write the claim first, then pick the hold-out axis. The general principle, that evaluation data is held out by site, operator, object or time as the claim requires and never mixed back into training, is set out in training vs evaluation datasets.
| Claim | Hold out by | Why |
|---|---|---|
| Works on future production | Time period and lot | Materials, tooling wear and recipes drift |
| Transfers to another line or plant | Line, plant, camera, lighting recipe | Models often degrade across lines; see domain shift across lines |
| Handles new part numbers or variants | Part number | A model can key on a part's geometry rather than its defects |
| Survives an imaging change | Camera, lens or lighting setup | Appearance shifts without any change to the part |
| Works from a humanoid's view | A separate robot-captured stratum | Fixed station views differ from head or wrist views |
| Catches a rare class | Distinct defective parts per class | Rarity limits what a test can show; see rare defects and class imbalance |
Checklist
Thresholds are yours to set. The "pass if" column says what a passing answer looks like, and the last column names the evidence to request.
Independence and custody
| Check | Pass if | Evidence to request |
|---|---|---|
| Split key | No part serial, lot or near-duplicate frame appears in both training and evaluation data | Split manifest keyed on serial and lot; duplicate report; see leakage in defect datasets |
| Tuning isolation | No threshold, augmentation or model choice was tuned on the evaluation set | Written protocol; access log |
| Label custody | Labels are held by a party that does not train the model, or the set is scored through a server | Description of who holds labels and how scores are returned |
| Process match | Evaluation parts come from the production process the model will face | Provenance flag per part: production, seeded or synthetic |
MVTec AD 2 shows one custody pattern: it keeps a private test set that is scored through an evaluation server (Heckler-Kram et al., 2025).
Coverage
| Check | Pass if | Evidence to request |
|---|---|---|
| Class coverage | Every class named in the claim has at least the number of distinct defective parts and lots you set | Per-class table of images, instances, parts and lots |
| Nominal coverage | Nominal parts span lots, shifts, lines and lighting | Nominal counts per stratum |
| Borderline parts | Near-threshold parts are present and flagged, not discarded | Count and examples; see borderline samples |
| Hard cases | Strata exist for the hard cases relevant to your parts | Stratum list with counts |
| Defect kinds | Structural flaws and logical ones such as missing or misplaced components are both represented where relevant | Taxonomy with example images |
Public benchmarks suggest which hard-case strata to ask for. MVTec AD 2 contains eight scenarios and more than 8,000 high-resolution images, adding small defects in large images, defects at the image border, high normal variability, transparent and reflective objects, overlapping bulk goods and a test split captured under varied lighting (Heckler-Kram et al., 2025). MVTec LOCO separates structural anomalies such as scratches and dents from logical ones such as missing or misplaced objects (Bergmann et al., 2022); for the logical side on assemblies, see assembly completeness verification.
Labels
| Check | Pass if | Evidence to request |
|---|---|---|
| Gold-label process | Two reviewers label each evaluation part independently and a third adjudicates disagreements | Guidelines, reviewer qualifications, adjudication log |
| Label basis | Defect presence, as an expert sees it, is recorded separately from the plant's disposition | Schema; see inspection images linked to outcomes |
| Localization | If you claim localization, masks or boxes were verified on every evaluation defect | Mask QA report. MVTec AD, for comparison, supplies pixel-precise ground truth for its test anomalies (Bergmann et al., 2021) |
| Agreement | Agreement is reported per class | Agreement table; see inter-inspector agreement |
| Frozen guidelines | The taxonomy version is frozen before models are scored | Version and date; no relabeling after results |
Conditions and reporting
| Check | Pass if | Evidence to request |
|---|---|---|
| Imaging metadata | Camera, optics, lighting and recipe are recorded per image | Metadata dictionary; see imaging conditions metadata |
| Drift record | Dated changes to tooling, recipes and cameras are listed | Change log |
| Operating point | The threshold is fixed from validation data before scoring, with a stated cost ratio between misses and false rejects | Written protocol |
| Per-class results | Results are given per class with distinct-part counts and intervals, not as pooled accuracy | Report template |
| False-reject rate | Reported on nominal parts per stratum | Nominal results by line and lighting |
Rights and reuse
| Check | Pass if | Evidence to request |
|---|---|---|
| Evaluation license | The set is licensed for evaluation, with explicit terms on whether it may be used for training | License language; see licensing terms for robotics training data |
| Public benchmark use | You have checked whether a public set may be used as a commercial acceptance test | License text. MVTec AD (Bergmann et al., 2021) and MVTec AD 2 (Heckler-Kram et al., 2025) are licensed CC BY-NC-SA 4.0, a non-commercial license. This is not legal advice |
| Confidentiality | Customer designs and part markings are cleared or excluded | See confidentiality in defect data |
Illustrative evaluation set manifest
# Illustrative only — synthetic identifiers; the buyer sets every count and threshold
eval_set: pv_eval_v0
claim: "Escape rate for crack_edge and missing_thread on line L3, new part numbers, 2026 Q2"
held_out_by:
line: L3
part_number: [PN-0000-D, PN-0000-E]
period: "2026-04-01 to 2026-06-30"
strata:
nominal: {parts: "buyer-set", lots: "buyer-set"}
borderline: {flagged_by: "2 of 3 reviewers"}
defects:
crack_edge: {distinct_parts: "buyer-set", lots: "buyer-set"}
missing_thread: {distinct_parts: "buyer-set", lots: "buyer-set"}
scratch_cosmetic: {distinct_parts: "buyer-set", lots: "buyer-set"}
robot_view: {camera: head_rgb, purpose: viewpoint stratum, separate_from_station_views: true}
labels: {reviewers: 2, adjudication: third_reviewer, masks: all_defects, guideline_version: tax_v4_frozen}
operating_point: {threshold_source: validation_split, cost_ratio_miss_to_false_reject: "buyer-set"}
custody: {labels_held_by: eval_lead_outside_training_team, access_log: required}
license: {use: evaluation_only, training_use: prohibited, redistribution: prohibited}
Red flags
- The evaluation set was drawn at random from the same pool as the training data.
- Sizes are quoted in images only, with no counts of distinct parts or lots.
- Labels came from the very camera system being evaluated, so the test is circular.
- Defects are synthetic or seeded and not flagged as such.
- Everything comes from one lot, one line or one week.
- Guidelines were revised after model results were seen.
- No nominal parts, or nominal parts shot under different conditions from the defects.
- Results are offered as one pooled accuracy figure.
What an evaluation set cannot show
An evaluation set measures detection on the parts and conditions it contains. It says nothing about handling a part, and nothing about the next line or the next camera unless a stratum covers it. A good score on a public benchmark does not stand in for your parts. Evaluation sets also age: when tooling, suppliers or acceptance standards change, a set that was held out is still valid for the old process only. For how the training side differs, see defect image datasets.
Questions to ask suppliers
- Can you provide parts or images that have never been used to train or tune any model, and how do you know?
- How many distinct defective parts and lots sit behind each class?
- Who labeled the evaluation parts, were reviews independent, and is there an adjudication record?
- Can labels be delivered separately from images, or held back for scoring?
- Are nominal and borderline parts included with the same metadata as defects?
- Are physical rejects retained so labels can be re-checked?
- What changed on the line during the period: tooling, recipe, camera or lighting?
- May the set be used for evaluation only, and for how long?
Discuss your requirements
If you are specifying an inspection test, tell SourceX which defects, parts and conditions your evaluation set needs to cover. SourceX looks for US businesses that hold suitable inspection data, checks the data and the supplier's licensing permissions, and coordinates pricing, allowed uses and delivery in a license. SourceX's role is sourcing, rights review, licensing and delivery coordination; it can also arrange annotation and engage specialist capture partners, scoped case by case. SourceX reviews each request and follows up about potential sources; a request does not guarantee a match.
See also the inspection, defect and failure hub, the humanoid training data guide, the glossary entries for held-out data and eval sets, and the page on private evaluation sets.
Tell SourceX what your evaluation set needs to cover
Sources
- The MVTec AD 2 Dataset: Advanced Scenarios for Unsupervised Anomaly Detection — hard-case scenarios, varied lighting and a privately scored test set.
- The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection — pixel-precise test ground truth and a non-commercial license.
- Beyond Dents and Scratches: Logical Constraints in Unsupervised Anomaly Detection and Localization — structural versus logical anomalies.