Privacy and preparation
How to measure dataset acceptance failures: a failure report protocol
By SourceX Editorial · Updated
10 sections
Short answer
A dataset acceptance failure study treats every acceptance decision on a delivered dataset as one case, records whether it passed, codes each failure against a fixed defect taxonomy, and reports failure and cure rates with uncertainty. This page publishes the method only: the taxonomy, the population and sampling rules, the record fields and a pre-registration checklist. No results are reported.
Checklist
Dataset acceptance failure
Key takeaways
- Treat each acceptance decision on a delivery as one case, and count passes as well as failures so there is a denominator.
- Separate the breached criterion, the underlying defect and the stage at which it was detected.
- Code every failure against a frozen nine-code taxonomy with two independent coders and an adjudicator.
- Report every proportion with its count and an interval, and keep comparisons descriptive.
- Pre-register the plan before collecting, and publish all pre-specified analyses; this page reports no results.
What would a dataset acceptance failure study actually measure?
It would measure how often delivered datasets fail the buyer's acceptance test, which defects caused each failure, and whether the defect was cured. The unit of analysis is one acceptance decision on one delivery, not one dataset and not one vendor, because a single dataset can be submitted, rejected and resubmitted several times.
This page is a protocol. It describes how to collect and code acceptance incidents so that a later report can be checked by anyone who reads it. It does not report findings, and any figure that appears in the worked illustration further down is labelled illustrative and invented for teaching purposes.
The question is narrower than the familiar one about how to run acceptance testing. Contract windows, rejection notices and cure periods are covered in existing guides on acceptance procedure. The question here is empirical: across many acceptance events, what tends to go wrong, how is it caught, and what does that imply for how suppliers prepare and how buyers write criteria.
A good study separates three things that informal post-mortems blur together: the criterion that was breached, the underlying defect that breached it, and the stage at which someone noticed. A delivery can fail a row-count criterion because of a filter applied during export, and that filter is the defect worth counting.
Which acceptance events belong in the study population?
Include every formal acceptance decision in the study window, passes as well as failures. A collection that holds only rejections cannot produce a failure rate, because it has no denominator, and it will overstate whatever defect types happen to generate loud disputes.
Define the population before collecting anything: the organizations contributing cases, the period, the data types in scope and what counts as a formal decision. A decision is formal when a named person accepted, conditionally accepted or rejected a delivery against written criteria. Informal comments on a sample before a contract exist are a separate, earlier stage and should be logged as a different event type.
Sampling should follow the contributing organizations' complete acceptance logs where possible. If a contributor can only share some events, record how they were chosen. Convenience samples of memorable failures are allowed as a pilot, but they must be reported as such and kept out of any rate calculation.
- Inclusion: a written acceptance criterion existed before delivery, and a decision was recorded with a date.
- Inclusion: the delivery was a licensed or commissioned dataset, including resubmissions after a rejection.
- Exclusion: internal data moves with no buyer and supplier on either side.
- Exclusion: events where the criterion was written after the delivery arrived, which are logged separately because they test a different thing.
- Record for every contributor: how many acceptance events existed in the window and how many were shared.
The anonymized defect taxonomy
Every failure gets one primary defect code and any number of contributing codes from the table below. The primary code is the defect without which the delivery would have passed. The detection stage records when the defect became visible, which matters because defects caught at intake tell a different story from defects found weeks later in model training.
The taxonomy borrows its documentation categories from published dataset documentation practice. Datasheets for Datasets organizes documentation around motivation, composition, collection process, preprocessing, uses, distribution and maintenance, and several codes below map to gaps in those areas.
When two defects both breached criteria, coders pick as primary the one that appeared earliest in the supplier's pipeline, because fixing it would often have prevented the other. A masking step that ran on a truncated export, for example, makes the truncation primary. Write this tie-break rule into the codebook so that it is applied the same way every time.
| Code | Defect class | Definition for coders | Typical evidence in the incident file |
|---|---|---|---|
| SC | Scope mismatch | Records fall outside the agreed population, period, language or record type | Acceptance memo citing out-of-scope samples; scope clause |
| SF | Schema or format | Fields, types, encodings or file structure differ from the written specification | Validator output; schema diff against the spec version |
| CV | Coverage or volume shortfall | Fewer records, fields or time periods than specified, or a key segment missing | Row counts by segment versus the specification |
| DL | Duplication or leakage | Duplicate records, or overlap between splits the buyer asked to keep apart | Hash or near-duplicate report; split membership lists |
| LB | Label or annotation error | Labels, outcomes or annotations wrong at a rate above the agreed threshold | Audit sample with adjudicated labels |
| PI | Residual personal or confidential data | Identifiers or confidential values remain after the agreed removal or masking | Residual audit findings; redaction log |
| RP | Rights or provenance gap | Missing or inconsistent evidence of the right to license, or unclear source lineage | Provenance record; rights review notes |
| DC | Documentation gap | Data card, field dictionary or collection notes missing or inaccurate | Delivered documentation versus the agreed list |
| IN | Integrity or transfer fault | Corrupt, truncated or unverifiable files, failed checksums, unreadable archives | Checksum log; transfer records |
How should each incident be recorded so it can be shared safely?
Record each acceptance event as one row with fixed fields, then anonymize before the row leaves the contributing organization. The aim is a dataset that another analyst could recode and recompute without learning who bought or sold what.
Replace names with stable random identifiers, band anything that could identify a deal, and suppress small cells. A rule such as not publishing any breakdown cell built from fewer than a pre-set number of events should be written into the protocol in advance, with the number chosen before data arrive.
- Event ID, contributor ID (random) and delivery round number.
- Data type (for example ticket histories, transaction records, documents, audio) and delivery size band.
- Acceptance criteria present: list of criterion types written before delivery.
- Decision: accepted, conditionally accepted or rejected, with the decision date expressed as days since delivery.
- Primary defect code, contributing codes and the criterion each one breached.
- Detection stage: automated intake check, manual sample review, downstream use, or third-party audit.
- Cure outcome: cured on resubmission, cured by partial acceptance, waived, or unresolved, with rounds to resolution.
- Coder IDs for both independent coders and the adjudicated final code.
How do two coders reach the same defect code?
Give two trained coders the same incident file, have them code independently, measure agreement, and adjudicate disagreements with a third reviewer. Publish the agreement statistic alongside any result, because a taxonomy that coders cannot apply consistently produces counts that mean little.
Run a pilot on a small set of incidents that will not be used in the final analysis. Revise definitions where coders disagree, freeze the codebook with a version number, and do not change it during the main collection. If a new defect type appears mid-study, log it under an 'other' code with a free-text note and handle it in a declared exploratory analysis.
Coders should work from documents, not recollections: the written criteria, the delivery manifest, the validator output, the rejection notice and the resubmission record. Interviews can explain context but should not override the paper trail when the two disagree.
Which metrics and comparisons would make a result defensible?
Report proportions with their denominators and intervals, and keep comparisons descriptive unless the design supports more. The core metric is the share of acceptance events that ended in rejection or conditional acceptance on first submission. Secondary metrics describe what failed and what happened next.
At a glance
Which metrics and comparisons would make a result defensible
For each proportion, report a confidence interval suited to small counts. The NIST engineering statistics handbook recommends the Wilson interval for a binomial proportion and describes an exact method for small samples. A failure share computed from a handful of events should be shown with its wide interval, not as a bare percentage.
Comparisons worth pre-specifying include data type, whether the buyer accepted a sample before contracting, and whether criteria were measurable (a stated threshold) or descriptive (a phrase such as 'high quality'). These are associations. Contributors who write measurable criteria may differ in many other ways, so the protocol must not claim that criteria style causes fewer failures.
Decide in advance how resubmissions enter the denominators. The cleanest approach reports first-submission metrics on first submissions only, and reports resubmission outcomes as a separate cure analysis linked by event ID. Mixing rounds into one pool inflates the number of events and makes suppliers who resubmit often look worse than those who abandon a delivery after one rejection.
- First-submission failure share: rejected or conditionally accepted events divided by all first-submission events.
- Defect mix: share of failures by primary code, with contributing codes reported separately.
- Detection stage mix: share of failures found at intake versus later use.
- Cure share and median rounds to resolution among failures.
- Uncertainty: an interval for every proportion and the count behind it.
What could this design never conclude?
It cannot estimate the defect rate of datasets in general. The population is acceptance events at contributing organizations, and organizations that keep good acceptance logs probably differ from those that do not. Results describe that population and nothing wider.
It also cannot see defects that acceptance missed. Published work on benchmark test sets shows how easily this happens: Northcutt and colleagues estimated that test sets of ten widely used datasets contain label errors at an average rate of at least 3.3 percent, after finding candidates with an algorithm and confirming them with human reviewers. Their figure is a lower bound for those public benchmarks and says nothing about licensed business datasets, but it illustrates why a passed acceptance test is not the same as a clean dataset. The NIST/SEMATECH engineering statistics handbook makes the same point about acceptance sampling: its purpose is to decide whether a lot is likely acceptable, not to estimate the lot's quality.
Finally, the study cannot rank vendors or buyers, and it should not try. Small cells, anonymization and selection effects make such rankings misleading, and publishing them would discourage the honest contributions the study depends on.
Reporting bias deserves its own line in the limits section. Contributors decide what to share, and an organization embarrassed by a rights or privacy failure may quietly withhold those events. Asking each contributor for its total event count in the window, and comparing it with what was shared, is the only check the design has on this.
What must the registered plan fix before any delivery is coded?
Write the plan down, time-stamp it in a public or shared registry, and only then start collecting. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan before data collection or analysis, which lets readers separate planned analyses from exploratory ones. For reporting, the STROBE checklists for observational studies are a useful model for what a methods section should disclose.
At a glance
What must the registered plan fix before any delivery is coded
- Research question in one sentence, with the unit of analysis named as the acceptance event.
- Population, study window, contributor recruitment method and the share of each contributor's events shared.
- Inclusion and exclusion rules, including how resubmissions are linked to the original event.
- Codebook version, coder training, agreement statistic and adjudication rule.
- Primary metric, secondary metrics and every pre-specified comparison.
- Interval method for proportions and the minimum cell size for publication.
- Anonymization steps and who holds the key linking random IDs to real parties.
- Known limits stated in advance, and a promise to report all pre-specified analyses whatever they show.
- Exploratory analyses labelled as exploratory in the final report.
Worked illustration: coding one rejected delivery
Illustrative only; the delivery, counts and durations are invented. A buyer receives a support-ticket dataset whose criteria require every ticket to include its full message thread and resolution code, and set a residual-identifier threshold. The intake validator passes the schema check. A manual sample review of 200 tickets finds 23 with truncated threads and 2 with customer phone numbers in free text.
Coder A assigns CV (threads incomplete) as primary and PI as contributing. Coder B assigns SF as primary, reading the truncation as a field-length limit in the export format. The adjudicator checks the export log, finds that a character limit in the export tool cut long threads, and records SF as primary, CV as contributing and PI as contributing. Detection stage is manual sample review.
The supplier re-exports without the limit and reruns masking. The resubmission passes 9 days after the first rejection, so the event is coded as cured on resubmission in one additional round. In the study dataset this produces two linked rows: a first-submission failure and a second-round acceptance.
What suppliers can take from the protocol before they deliver
Even before any results exist, the taxonomy works as a pre-delivery checklist: run the same nine codes against your own export and fix what you find. Suppliers who keep their own acceptance log in this shape also gain a record of how each problem was cured.
Preparation and delivery are part of what SourceX does when a business licenses selected records as a dataset, so the nine codes are a natural shared language for an acceptance discussion. A delivery under its standing terms follows a signed agreement in which the business accepted the scope and terms, and the de-identification step that PI codes test sits with SourceX through approved processing routes. A business can learn whether it qualifies from a short assessment, without uploading anything.
Frequently asked questions
What is a dataset acceptance failure report?
It is a written record of why a delivered dataset did not pass the buyer's acceptance criteria: which criterion was breached, which defect caused it, when it was detected and how it was cured. Collected consistently across many deliveries, these reports become the raw material for an incident study.
Why count passed deliveries in a failure study?
Without passes there is no denominator, so you cannot compute how often failures happen. A file of rejections alone also overrepresents defects that cause visible disputes and hides those that are routinely caught and fixed quietly.
Does this page report how often datasets fail acceptance?
No. It publishes the method only: population rules, the defect taxonomy, record fields, metrics and a pre-registration checklist. Any numbers in the worked illustration are invented to show how coding works.
How many acceptance events does a study need?
Enough that the intervals around the main proportions are narrow enough to be useful, which depends on how common failures turn out to be. Pre-specify the interval method and a minimum cell size, and publish wide intervals honestly rather than dropping them.
Can contributors stay anonymous?
Yes. Names are replaced by random identifiers before rows leave the contributing organization, deal-identifying values are banded, and small breakdown cells are suppressed. Only a named custodian holds the key, and the protocol states who that is.
Sources
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process, recommended uses and related lifecycle areas. Source
- Northcutt, Athalye and Mueller estimate label errors averaging at least 3.3 percent across the test sets of ten widely used datasets, found with confident learning and confirmed by human reviewers. Source
- The STROBE initiative publishes reporting checklists for cohort, case-control and cross-sectional observational studies. Source
- The NIST/SEMATECH e-Handbook describes acceptance sampling as deciding whether a lot is likely acceptable, not estimating its quality. Source
- The NIST/SEMATECH e-Handbook recommends the Wilson interval for a binomial proportion and covers exact intervals for small samples. Source
- The Center for Open Science defines preregistration as posting a time-stamped, read-only study plan before data collection or analysis. Source
- Hugging Face documents dataset cards as the README that explains a dataset's contents, context, potential biases and metadata such as license, language and size. Source
See if your company qualifies
A short company assessment. No data uploads are needed.