Short answer
Borderline samples sit just either side of an acceptance limit, where inspectors disagree and models fail. Archives under-represent them, because most parts are clearly good or clearly bad. A specification should define the limit, select parts by measured distance from it, image them under controlled conditions, collect several independent labels and keep the set as an evaluation slice.
Why the boundary needs its own collection
Production data is lopsided. Most parts pass without question, a few are obvious scrap, and a thin band in between is where inspectors hesitate. A model can score very well on a random sample by getting the extremes right while failing in the band, which is where the commercial cost of errors tends to sit: scrap that was fine, or escapes that were not.
A purpose-built borderline set lets a buyer test whether the model's decision boundary lies where the standard puts it. It also exposes the effect of imaging conditions on what is visible near the limit. This is a narrow collection with high value per image. It complements, and does not replace, the broader archives described in sourcing defect image datasets and the class problems in rare defects and class imbalance.
First define the limit
A borderline set is meaningless without a stated threshold. The threshold comes from a standard, a drawing or a customer agreement: a scratch length, a void fraction, a solder fillet condition. Standards frequently differ by product class. IPC-A-610 defines Class 1 general, Class 2 dedicated service and Class 3 high performance, the strictest (IPC, 2024); the same assembly condition can sit on different sides of the line depending on the class. So the spec must name the standard, revision and class, as covered in acceptance standards as label definitions.
Where the criterion is a measurement, define how it is measured. Where it is visual, supply reference images; see golden samples and reference images.
What a borderline set can and cannot support
It can support: evaluation of boundary accuracy; threshold calibration; hard-example training or fine-tuning; analysis of where inspectors disagree; and testing of imaging-condition sensitivity.
It cannot support, on its own:
- General defect detection training. It is deliberately unrepresentative.
- Prevalence estimates. Nothing in a selected set tells you how often borderline parts occur.
- A standard. The labels reflect the raters' reading of a standard, not the standard.
- Anything about manipulation. Handling the part is a separate task needing robot data.
Using it as training data is a design choice that removes it as an evaluation set. Keep a separate adjudicated slice for evaluation, held out by lot or line.
Illustrative collection specification
All values are synthetic and set by the buyer.
# Illustrative: borderline-sample collection spec
standard: {name: "customer_spec", revision: "C", product_class: "class_2"}
criterion: {feature: "edge scratch length", limit_mm: 10.0}
selection:
band_mm: [7.0, 13.0] # measured distance from limit on both sides
strata:
- {range_mm: [7.0, 9.0], target_share: 0.25}
- {range_mm: [9.0, 10.0], target_share: 0.25}
- {range_mm: [10.0, 11.0], target_share: 0.25}
- {range_mm: [11.0, 13.0], target_share: 0.25}
per_line: [line_A, line_B]
measurement: {method: "calibrated microscope", repeat: 2, tolerance_mm: 0.2}
imaging:
views: [top, raking_light_30deg, side]
lighting_log: required
resolution: "sufficient to resolve 0.1 mm"
scale_reference: "in-frame ruler"
labels:
raters: 3 # independent, blind to measurement
fields: [defect_type, severity, disposition, confidence]
adjudication: "senior quality engineer; rule written"
metadata: [lot, line, date, part_revision, inspector_role]
The selection band and strata are designed to give coverage on both sides of the limit. The measurement fields give a numeric ground truth separate from human opinion. Raters stay blind to that measurement to produce an honest agreement figure.
Where borderline samples come from
- Production retains. Parts held for review, rework bins and first-article sets often contain near-threshold items. Quality teams can pull them if the plant keeps them.
- Measurement records. Dimensional inspection records identify parts measured near limits, which can be re-imaged.
- Physical retention. If parts survive, new imaging under controlled conditions is possible; see physical defect samples.
- Archive mining. Parts with disagreement between inspectors or between machine-vision and human calls.
- Commissioned production. Deliberately making parts near the limit, where the process allows and the customer agrees.
Existing archives provide images captured for production, with whatever lighting the line had. A new imaging campaign controls conditions but needs site access, parts that were kept and rights to image them.
Quality and acceptance tests
- Band coverage. Counts per stratum within agreed tolerances.
- Measurement repeatability. Repeat measurements within the stated tolerance.
- Label agreement. Report agreement by stratum; lower agreement near the limit is expected and informative. Methods are in inter-inspector agreement and label noise.
- Imaging sufficiency. A sample of images must let a reviewer see the feature at the threshold scale.
- Metadata completeness. Lot, line and part revision present for every image.
- Leakage control. No sibling images of the same part spread across splits.
- Separation. The evaluation slice is tagged and kept out of training.
Limitations and additional preparation
Mixed supervision research suggests detailed labels can be targeted at a subset rather than everything (Božič et al., 2021), which is helpful for budgeting a small borderline set, but borderline images are exactly the ones where pixel-accurate masks are hardest to agree on. Plan for adjudication time.
Selection by measurement may not match how the defect looks. A shallow wide scratch and a short deep one can share a measured length but look different. If the standard is visual, an honest borderline set should include both. Parts held for review may also be biased towards those a particular inspector found worrying.
Customer parts raise confidentiality questions; see confidentiality in defect data.
Working with the set after collection
Report results on the borderline set by distance from the limit. A curve of accuracy against measured distance shows where the model's decision boundary lies and whether it is shifted toward over-rejecting or over-accepting. This is more informative than a single score and can be compared with human agreement across the same strata.
Be careful when moving borderline samples into training. A common pattern is to split by part, using some for fine-tuning and keeping others for evaluation. Keep multiple images of the same part on one side of the split. Keep a final slice of untouched, adjudicated parts that no one tunes against.
The measurement record is also a label source. For parts with measurements, the model can be trained or evaluated on predicting the measured value and not only the pass/fail call, which gives a continuous target near the limit. That helps when the standard changes: the same measured parts can be re-labeled against a new limit without re-imaging.
Questions to ask suppliers
- Does your plant keep parts held for review, and for how long?
- How are near-limit parts identified: by measurement, by inspector call or by disagreement?
- Can parts be re-imaged under controlled lighting, and by whom?
- Which standard, revision and product class apply to each part family?
- Are measurement records available, with instrument and calibration details?
- How many independent inspectors can label each part, and are they blind to the measurement?
- Are the parts customer-designed, and what permission is required to image and license them?
- Can you deliver a small pilot set so the band and strata can be tested before scale-up?
Discuss your requirements
If decision-boundary errors are the risk, tell SourceX what your evaluation set needs to cover, including the standard, the threshold and the lines to include. SourceX looks for US businesses that hold suitable material and follows up about potential sources; if no existing asset fits, SourceX can scope a commissioned collection. A request does not guarantee a match. See the humanoid training data guide and the inspection, defect and failure data hub for context, and the page on defect severity and disposition labels for the label fields.
Tell SourceX what your evaluation set needs to cover
Sources
- IPC-A-610 Acceptability of Electronic Assemblies — product classes with different acceptance strictness.
- Mixed supervision for surface-defect detection: from weakly to fully supervised learning — targeted detailed labels at lower annotation cost.