Skip to content

Privacy and preparation

Dataset batch acceptance sampling or full inspection: how to choose

By SourceX Editorial · Updated

9 sections

Short answer

Use dataset batch acceptance sampling only for checks that need human judgment, such as label correctness or redaction quality. Run every machine-checkable test on all records, including schema, counts, timestamps, attachment hashes and scope reconciliation, because a full check costs little more than a sampled one. Treat defects that can never be tolerated, such as a leaked identifier class, as full-screen items, since a clean sample cannot prove their absence.

Checklist

Dataset batch acceptance

This page covers dataset batch acceptance sampling or full inspection: how to choose.Illustrative

Key takeaways

  • Decide inspection mode per check, not per batch: every record, a random sample, or a full screen.
  • Machine checks such as schema, counts, timestamps and hashes should run on every record because they are cheap and exhaustive.
  • Sampling fits judgment checks, and a clean sample only bounds the defect rate; it never proves zero.
  • Define a lot as one export from one system under one schema version, so a rejection points to a fixable cause.
  • Confirmed identifiers, rights mismatches or systematic label drift end sampling and go to a specialist.

Which checks belong on every record, and which on a sample?

Run a check on every record whenever software can run it; sample only the checks that need a person to judge. A dataset batch is not a pallet of parts, and the reasons manufacturers sample rarely apply to records that a script can read in minutes.

The NIST/SEMATECH engineering statistics handbook lists the conditions under which acceptance sampling is used: testing is destructive, full inspection is very costly, or full inspection takes too long. Schema validation, row counts and timestamp parsing meet none of them. A validator does not damage records, and checking a million rows usually costs about as much engineering as checking a thousand.

Human review is different. Deciding whether a ticket was routed to the right queue, whether a redaction left a name in free text, or whether an outcome label matches the case history takes minutes per record. Those checks are costly and slow at full volume, which is exactly where sampling earns its place.

The handbook also makes a point buyers and suppliers often forget: acceptance sampling exists to decide whether a lot is likely acceptable, not to estimate how good it is. A batch that passes a sampled review has been accepted under a stated risk, not certified clean.

Think of the decision as three questions asked of each check. Can software decide it without a person? Then run it on every record. Does it need judgment, and would a small residual rate be tolerable? Then sample it. Would even one occurrence be unacceptable, as with a leaked identifier? Then screen every record automatically and have a person review every hit.

Why a batch can open cleanly and still fail

Checking that every file opens is the weakest possible acceptance test, because most serious defects in business data live inside readable files. A batch can parse perfectly and still cover the wrong date range, duplicate a month of records, or reference attachments that never shipped.

At a glance

Why a batch can open cleanly and still fail

Tools built for data pipelines already treat checks as queries that return failing records. dbt's documentation describes data tests as select statements that look for records disproving an assertion; a test passes when it returns zero failing rows, and the built-in generic tests cover non-null values, unique values, accepted values and relationships between tables. That is the right shape for batch acceptance: a list of assertions, each run across the whole batch, each producing the failing records as evidence.

The defects listed below commonly survive a readability check. Each is invisible to someone who opens a few files and scrolls, and each is caught in seconds by a query across the whole batch, which is the practical argument for running them everywhere: the cost of the full check is the cost of writing it once.

  • Scope drift: records outside the agreed period, business unit or record type, often from a filter applied in one export but not the next.
  • Count gaps: case totals that do not reconcile with the source system's own report for the same period.
  • Timestamp ambiguity: local times with no offset, so ordering across sites or daylight-saving changes is unreliable.
  • Schema drift: a renamed or repurposed field partway through the period, with no version marker.
  • Broken attachments: files referenced in records but missing, truncated or replaced in the package.
  • Outcome leakage or gaps: outcome labels missing for open cases, or filled from information that would not have existed at decision time.

The risk-based acceptance sampling plan

The plan assigns each check one of three modes. Every record means an automated test across the full batch. Sample means a random draw reviewed by a person under a stated acceptance rule. Full screen means an automated search across every record for a zero-tolerance defect, with every hit reviewed by a person.

Agree the plan with the buyer before the first delivery, then run it yourself before shipping. Each check names the evidence kept, so a disputed rejection can be settled from files rather than recollection.

Write the defect definition so that two reviewers would reach the same verdict on the same record. A definition such as label looks wrong invites argument; a definition such as resolution category differs from the category recorded at case closure can be checked and disputed on the facts.

Keep the random seed and the drawn record IDs for every sample. If the buyer repeats the draw with the same seed and gets different records, the lot has changed since it was sampled, which is itself a finding.

The risk-based acceptance sampling plan
CheckDefect definitionModeAcceptance ruleEvidence kept
Scope reconciliationRecord outside agreed period, unit or typeEvery recordZero out-of-scope recordsQuery, counts by period and unit
Row and case totalsTotals differ from the source system reportEvery recordDifference explained in the exception logSource report export and reconciliation sheet
Timestamp validityUnparseable, offset missing, or out of order within a caseEvery recordZero unparseable; offsets present on allValidator output with failing IDs
Schema and version historyField missing, renamed or retyped without a version noteEvery recordAll records match a declared schema versionSchema files and version changelog
Attachment integrityReferenced file missing or hash mismatchEvery recordZero missing; all hashes match the manifestManifest with SHA-256 values
Outcome labelsLabel absent, inconsistent with the case history, or written after the cutoffEvery record for presence; sample for correctnessPresence 100%; correctness within the agreed limitCoverage report and reviewer sheets
Direct identifiers after de-identificationAny listed identifier class presentFull screenZero confirmed hitsScreen patterns, hit list and dispositions
Free-text redaction qualityName or contact detail left in narrative textSampleAgreed zero-defect sample size passedSample IDs, seed and reviewer notes
Exception logKnown issue not disclosed with the batchEvery recordEvery excluded or flagged record listed with a reasonException log shipped with the batch

How large should the judgment sample be?

Size the sample from the defect rate you need to rule out, not from a round number. If a reviewer finds no defects in n randomly drawn records, the data support only the claim that the batch's defect rate is probably below a bound that shrinks as n grows.

The bound comes from the exact binomial confidence limits described in the NIST handbook. With zero defects in n records, the one-sided 95% upper limit is 1 minus 0.05 raised to the power 1/n, which is close to 3 divided by n. The table below applies that formula; it is arithmetic, not a standard, and assumes records are drawn at random from the whole batch.

Two consequences follow. A clean review of 100 records only supports a defect rate below about 3%, which is rarely what a buyer means by clean. And driving the bound low enough for a rare, serious defect takes thousands of reviewed records, which is why such defects belong in the full-screen mode instead.

Agree the acceptance number with the buyer before sampling. Zero defects allowed in the sample is the simplest rule and the easiest to audit; allowing one or two defects needs a larger sample for the same protection, and the arithmetic for that is what the AQL tables encode.

How large should the judgment sample be?
Records reviewed, zero defects found95% upper bound on defect rateTypical use
60About 4.9%Early pilot sample, not acceptance
150About 2.0%Low-impact label fields
300About 1.0%Routine label correctness
600About 0.5%Labels a buyer trains decisions on
3,000About 0.1%Rarely practical by hand; consider a full screen

Defining a lot so a rejection means something

A lot should be one export from one source system, over one period, under one schema version. When a lot mixes sources, a failed sample cannot say which part caused the failure, and the whole batch goes back when only one export needed fixing.

Standard sampling schemes assume a coherent production stream. ISO 2859-1 specifies AQL-indexed plans for lot-by-lot inspection of a continuing series of lots, with switching rules that tighten or relax inspection as history accumulates. Those rules make sense for a recurring data feed from one system; they make little sense for a one-time archive, which should be treated as an isolated lot.

Make the sampling unit match the thing a model will learn from. If a support case spans a ticket, six messages and two attachments, the unit is the case, not the row, and the reviewer judges the case as a whole. Stratify the draw when the batch covers distinct periods or business units, so a problem confined to one quarter is not diluted by three clean ones.

Record the lot boundaries in a manifest that ships with the data: source system, export date, period covered, schema version, record count and a hash for each file. The manifest turns a rejection into a precise instruction, such as re-export lot two with the corrected filter, instead of a general complaint about quality.

Worked example (illustrative): a support archive in three batches

This example is illustrative; the counts are chosen to show the method, not taken from a real delivery. A software company prepares two years of support cases from its help desk for licensing, shipped as three lots, one per export run, each under its own schema version.

The every-record checks on lot one run in under an hour. They find 212 cases whose attachments are referenced but missing from the package, and 38 cases dated a week outside the agreed period. Both are fixed at the source and the lot is re-exported; none of this needed a reviewer.

The full screen for direct identifiers flags 1,140 candidate hits across 48,000 cases. Reviewers confirm 9 as real phone numbers in signature blocks, and the team traces them to one redaction rule that skipped a number format. The rule is fixed, the screen is rerun with zero confirmed hits, and the fix goes into the exception log.

For label correctness, reviewers check 300 randomly drawn cases for whether the resolution category matches the case history and find 4 defects. That exceeds the agreed rule of zero defects in 300, so the team examines the 4, finds they all involve one retired category, remaps it across the lot, and draws a fresh sample of 300, which passes.

Lots two and three run the same plan. Because lot one's problems were fixed at the source, both pass their every-record checks first time, which is the practical payoff of running the buyer's plan before shipping.

When inspecting every record by hand is the right call

Full manual review is justified when the batch is small, every record carries high stakes, or the defect cannot be found by software. An evaluation set of a few hundred gold-labelled cases is the common example: each label directly sets the score a model is judged by, and reviewing all of them costs less than arguing about a sampling plan.

Rejected lots are another case. The NIST handbook notes that a common procedure for non-destructive testing is to inspect rejected lots in full and replace the defective units. For data, that means a lot that fails its judgment sample is either reviewed in full for that check or fixed at the source and resampled, never simply resampled until a draw happens to pass.

The limit of this recommendation is cost. Manual review of a large narrative archive can take longer than the delivery window, and fatigue lowers reviewer accuracy over long sessions. When full review is required for a large batch, plan it as a staged project with its own quality checks on the reviewers.

Findings that end sampling and go to a specialist

Some findings change the question from whether to accept the lot to whether the lot can be licensed at all. Stop the acceptance process and route the finding when any of the following appears:

  • A confirmed direct identifier in data described as de-identified goes to the privacy reviewer, along with the rule that missed it.
  • Records from a business unit, client or system that the rights review did not cover go to counsel before anything else ships.
  • A label defect that follows a pattern, such as one retired category or one site's coding habit, goes to a domain expert to decide whether the labels can be remapped or must be removed.
  • A count difference the source system cannot explain goes back to the system owner, because it may mean records were lost or duplicated upstream.

Running the buyer's plan before you ship

Suppliers who run the buyer's acceptance plan themselves usually find the cheap problems, missing files and scope errors, before the buyer does. Ship the exception log, the manifest and the check outputs with every lot, so acceptance becomes a review of your evidence rather than a search for defects.

SourceX helps companies license selected records and runs rights review, preparation and delivery as part of that work. It does not host very large datasets: big deliveries stay on the company's own storage or ship on encrypted drives, so acceptance checks run where the data already lives. A short assessment settles whether the company qualifies before any record leaves its systems.

Frequently asked questions

What is dataset batch acceptance sampling?

It is reviewing a random sample of records from a delivered batch and accepting or rejecting the whole batch based on the defects found. It suits checks that need human judgment. Checks that software can run should cover every record instead.

Does a clean sample mean the batch has no defects?

No. Zero defects in a random sample only bounds the defect rate; with 300 clean records, the 95% upper bound is about 1%. Defects that can never be tolerated need a full automated screen across every record.

Should a supplier share its own acceptance results with the buyer?

Yes, with the evidence behind them: check outputs, the hash manifest, sample IDs with the random seed, and the exception log. The buyer will still run its own checks, but shared evidence shortens disputes and shows where defects were fixed at the source.

What should happen to a lot that fails its sample?

Find the cause in the failing records, fix it at the source where possible, and draw a fresh sample, or review the failed check across the whole lot. Repeatedly resampling until a draw passes defeats the purpose of the plan.

Is AQL sampling the right standard for data deliveries?

It fits recurring feeds from one system, where switching rules can tighten or relax inspection over many lots. For a one-time archive, treat the delivery as an isolated lot and size the judgment sample from the defect rate you need to rule out.

Sources

  • Acceptance sampling is used when testing is destructive or full inspection is too costly or slow, and it decides whether a lot is acceptable rather than estimating its quality. Source
  • dbt data tests are queries that return failing records, with built-in tests for non-null, unique, accepted values and relationships. Source
  • Exact binomial confidence limits give an interval for a proportion defective, including for small samples and few defects. Source
  • ISO 2859-1 specifies AQL-indexed sampling schemes for lot-by-lot inspection of a continuing series of lots, with switching rules. Source
  • RFC 3339 timestamps carry an explicit offset from UTC, either Z or a numeric offset. Source
  • Secure hash algorithms produce digests that change, with very high probability, when a message changes. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies