Skip to content

Definitions and comparisons

Enterprise dataset rejection taxonomy: six causes and who can cure each

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

An enterprise dataset rejection taxonomy gives every rejected delivery one primary code from six families, rights, relevance, format, duplication, quality and commercial, tested in that order and tied to the document it rests on. Code the first gate that fails, then choose the cure by who can fix it: data work, re-scoping, a contract change or nothing. A technical fix cannot cure rights, terms or a task mismatch.

Value drivers

Enterprise dataset rejection

This page covers enterprise dataset rejection taxonomy: six causes and who can cure each.Illustrative

Key takeaways

  • Code every rejected delivery with one primary family, rights, relevance, format, duplication, quality or commercial, and the document the code rests on.
  • Run the gates in a fixed order: what data work cannot fix first, then format before duplication, duplication before quality, and commercial terms last.
  • Reconcile the approved scope, the measured quality report and the buyer's acceptance test before choosing a cure; a broken test is not a data defect.
  • Cures cascade: redaction for rights can break relevance, and deduplication can push a delivery below its commercial terms.
  • Some rejections have no cure, and logging them as closed keeps the remediation backlog honest.

What should an enterprise dataset rejection taxonomy decide?

It should decide one thing per rejection: who can cure it. A taxonomy that only labels what went wrong produces tidy reports and the wrong remediation; one that ties each reason to its evidence and to the party able to fix it tells the acceptance owner whether to send a delivery back for data work, rewrite the scope, reopen the contract or close the file.

The six families in this guide are rights, relevance, format, duplication, quality and commercial. Every rejection gets exactly one primary family, chosen by a fixed test order, plus secondary codes for anything else that was actually tested and failed. Each code must point to one of three documents: the approved scope, the measured quality report or the buyer's acceptance test. A rejection that cannot cite one of them is not yet a rejection; it is an opinion waiting for evidence.

This page is for the person who owns acceptance on the buyer side, and for the delivery owner answering them. It starts after a delivery has failed review. Diagnosing a declined offer and repairing data after acceptance are separate jobs, covered in the SourceX guides linked from this page.

The six reason families and where each boundary sits

Each family answers a different question, and the boundary rules matter as much as the definitions, because most coding disputes are about which family a reason belongs to. Published standards and research anchor the families where they exist.

Rights evidence deserves suspicion even for public data: the MIT Media Lab's audit, covering more than 1,800 text datasets, showed popular dataset hosting sites omitting licenses at rates above 70 percent and listing them in error at rates above 50 percent. Enterprise records rarely carry a license field; permission sits in contracts and notices, which is why a rights code must cite the review notes rather than a label. For relevance, the EU AI Act's data governance article for high-risk systems offers a usable test: training, validation and testing data must be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Quality codes can follow ISO/IEC 25012, which defines fifteen data quality characteristics under inherent and system-dependent views.

Duplication earns its own family because it changes every other count. Lee and colleagues found near-duplicates making up 3.04 percent of C4 and 13.63 percent of RealNews under one detection method, and 4.6 percent of C4 validation examples duplicated in the training set. Format is the most mechanical family: OpenAI's fine-tuning data preparation notebook runs seven named format checks, from a missing message list to an unrecognized role, and counts each error type separately. Commercial reasons cover price, term, exclusivity and payment structure, judged against the scope that survives the other gates.

The six reason families and where each boundary sits
FamilyFails whenEvidence to citeBoundary ruleUsual cure
RightsThe intended use is not permitted, or permission cannot be shownRights review notes, license terms, privacy basis, third-party content logA usage restriction written into the terms is coded here, not as commercialContract change, or none
RelevanceRecords miss the task, population, period or language the buyer namedAcceptance test, task description, coverage counts by segmentToo few in-scope records is relevance; a fair price that the in-scope count cannot support is commercialRe-scope, or none
FormatFiles cannot be parsed or loaded as specifiedValidator output against the named schema versionReadable files with wrong values are qualityData work
DuplicationRecords repeat, nearly repeat, or overlap data the buyer already holds or tests onExact and near-duplicate report; overlap with named holdout setsLegitimate repeat events, such as two real orders, are not duplicatesData work, then recheck commercial
QualityValues, labels or links miss a measured thresholdQuality report version, audited sample, the ISO/IEC 25012 characteristicResidual personal data after agreed masking is coded under rightsData work
CommercialPrice, term, exclusivity or payment do not fit the usable scopeOffer terms set against final usable countsTerms that limit permitted use are coded under rightsContract change, or none

Why the test order runs rights, relevance, format, duplication, quality, commercial

Test first what data work cannot fix, then test in the order each measurement depends on the last. That order stops reviewers from spending a week on quality metrics for a delivery the buyer could never use.

Steps

Why the test order runs rights, relevance, format, duplication, quality, commercial

Rights and relevance come first because no cleaning, conversion or deduplication can cure them. Format comes next because nothing can be measured in a file that cannot be read. Duplication precedes quality because quality rates computed on duplicated records count the same defect, or the same clean record, many times over. Commercial comes last because a price can only be judged against the scope that survives the other five gates: the same price may be fair for the delivered count and unfair for the deduplicated one.

The primary code is the first gate that fails in this order. Later gates are coded only if they were actually run and failed, and gates never run are logged as untested rather than passed. That rule makes rejections comparable across reviewers and over time, which a free-text reason never is. Here is the same order written as a decision tree:

  • Gate 1, rights: is the intended use permitted for every record, and can that be shown in writing? If not, code rights and stop technical review.
  • Gate 2, relevance: do the records match the task, population, period and language in the acceptance test? If not, code relevance; re-scope when matching records exist, otherwise close.
  • Gate 3, format: does every file pass the agreed schema validator? If not, code format and return the files for conversion.
  • Gate 4, duplication: after exact and near-duplicate checks, and a check against the buyer's holdout sets, is the unique count still within scope? If not, code duplication, deduplicate, and flag gate 6 for a recheck.
  • Gate 5, quality: do measured values, labels and links meet each threshold on the deduplicated records? If not, code quality with the characteristic and the measured value.
  • Gate 6, commercial: do price, term and exclusivity fit the usable scope that remains? If not, code commercial and reopen terms.
  • All six pass: accept, and log the version of each document the gates relied on.
Six acceptance gates in a row, rights, relevance, format, duplication, quality and commercial, each with a failure exit to its usual cure, ending in accept.
The fixed test order. The primary code is the first gate that fails; the exit shows the usual cure.

Reconcile scope, measurement and the acceptance test before choosing a cure

Before any cure is chosen, line up the three documents a rejection can rest on: the approved scope, the measured quality report and the buyer-specific acceptance test. Where they disagree, the disagreement, not the data, is often the real problem.

Acceptance tests can be wrong. When OpenAI had 93 software developers experienced in Python review 1,699 random SWE-bench samples in 2024, they screened out tasks whose issue descriptions were underspecified or whose unit tests were too narrow to accept valid solutions, then released a 500-sample verified subset. Anthropic's guide to agent evaluations makes the same point from the grader's side: a task is sound when two domain experts, judging alone, would reach matching pass or fail calls, and a 0 percent pass rate across many trials with a frontier model most often signals a broken task. A delivery that fails a broken test has a test problem, and coding it as a quality rejection sends the supplier to fix the wrong thing.

The table maps the common mismatches to a code and a cure.

Reconcile scope, measurement and the acceptance test before choosing a cure
What the three documents showWhat it meansCodeCure
Scope allows it and measured quality meets the signed spec, yet the buyer test failsThe buyer's test asks for more than the agreed specificationCommercial, or relevance if the task itself changedRenegotiate the spec or the price; data work will not close it
The buyer test asks for records outside the approved scopeA scope change presented as a defectRelevanceAmend the scope by contract, then source; do not patch the delivery
Measured quality falls below the spec both sides signedA genuine defectQuality, or formatData work and redelivery
The buyer test fails even on reference examplesThe acceptance test is brokenNo dataset code; log as test under reviewFix the test with the buyer before recoding
Rights review notes do not cover the buyer's stated useA permission gap, whatever the data looks likeRightsContract change, or close

The rejection reason log: fields that make a rejection reproducible

Log every rejection in one record with enough detail that a second reviewer, given the same files, would assign the same code. If two reviewers disagree on more than an occasional case, fix the boundary rules, not the reviewers.

At a glance

The rejection reason log: fields that make a rejection reproducible

Version the log with the taxonomy, so codes assigned under one version can be compared with codes under the next. Have a second reviewer code a sample of rejections independently each quarter and record agreement, both raw and corrected for chance, before trusting any trend in rejection reasons. The fields that make this work:

  • rejection_id, dataset name, delivery version and review dates.
  • gate_reached and primary_family with a subcode, for example duplication: holdout overlap.
  • secondary_codes, each marked tested and failed, with untested gates listed separately.
  • document_cited: the scope clause, quality report version or acceptance test version the code rests on.
  • evidence_pointer: the record IDs or the query that selects them, with the count affected.
  • measured_value, threshold and denominator, so anyone can recompute the rate.
  • reviewer and second_reviewer, with an adjudication note if they differed.
  • cure_class (data work, re-scope, contract change or none), cure_owner and target date.
  • retest_result with the delivery version it ran on, then a final status: cured, closed or withdrawn.

When a technical fix cannot cure the rejection

A technical fix cures format failures, most duplication and many quality failures; it cannot cure rights, commercial terms or a genuine task mismatch, and trying can make matters worse. The cure column exists to stop that mistake before anyone spends engineering time.

Three cure cascades recur. Redaction applied to answer a rights concern can strip the free text a relevance test depends on, turning a rights rejection into a relevance rejection. Deduplication cures a duplication rejection but can shrink the unique count below what the commercial terms assumed, so every duplication cure should trigger a fresh look at gate 6. And cleaning records to pass a quality threshold cannot help when the records describe the wrong task: a perfectly clean dataset of the wrong decisions still fails.

The honest outcome for some rejections is no cure. When the permitted use does not cover the buyer's purpose, or when no records of the needed task exist, the right entry is cure class none, a closed status and a note of what a future request should specify. Logging such cases as open data work inflates the remediation backlog and hides the real constraint.

Worked example (Illustrative): coding one rejected claims-notes delivery

Illustrative only: no real company, buyer or dataset is described. A buyer's acceptance test for a claims-triage model asks for adjuster notes linked to final claim outcomes, from two named lines of business, in English, delivered as JSON Lines in a fixed schema. The delivery fails review with a one-line email saying the data quality is too low.

The acceptance owner runs the gates in order. Rights pass, because the review notes cover the stated use. Relevance passes for one line of business and fails for the other, where most notes lack a linked outcome; that is coded relevance with a coverage subcode, because the gap is missing in-scope records, not bad values. Format passes. The duplication check finds notes repeated across reopened claims and records them as a secondary failure with the affected record IDs. Quality runs only on the unique, in-scope records and passes. Commercial is flagged for a recheck against the smaller usable count.

The log therefore reads: primary relevance, cure class re-scope; secondary duplication, cure class data work; commercial flagged for recheck. The buyer's original complaint, data quality, appears nowhere as a code, because the measured quality report met its threshold. The supplier deduplicates, narrows the scope to the line of business with linked outcomes, and both sides agree a revised count before redelivery.

What the taxonomy cannot tell you

It classifies decisions; it does not measure how often each reason occurs across a market, and early counts from one buyer or one supplier say little about anyone else. SourceX has not published rejection counts or rates, and none are reported here.

Codes are judgments, even with boundary rules. Two reviewers can reasonably split on whether a thin segment is relevance or commercial, which is why the log records the evidence and the agreement rate instead of treating a code as fact. The taxonomy also simplifies interactions: one delivery can fail for reasons that cause each other, and the primary-code rule picks the first gate in the order, not necessarily the deepest cause. Revisit the boundary rules whenever a new reason appears in more than a couple of rejections, and version the taxonomy when you change them.

Nothing here is legal advice. Rights codes in particular depend on contracts, notices and the law that applies to the records, and counsel should review any rejection coded under rights before a cure is chosen.

Turning rejection codes into a sharper data request

Feed every closed rejection into the next request, so the same reason does not recur. A relevance rejection becomes a sharper population and period, a duplication rejection becomes a named holdout set the supplier must check against, and a format rejection becomes the schema file attached to the request.

SourceX does not stock datasets for AI buyers; it sources them from U.S. businesses on request, and the request-data form collects the buyer's timing, scope, volume, language, format and licensing needs. Adding the intended use and an acceptance test gives the six gates something concrete to run against from the start. On the supply side, the company providing the records signs only after approving the scope and terms, and nothing is delivered before that, so rights and scope questions surface before there is a delivery to reject.

Frequently asked questions

What are the main reasons enterprise AI datasets get rejected?

Most rejections fall into six families: rights, relevance, format, duplication, quality and commercial. They differ in who can fix them. Format, duplication and most quality failures yield to data work, while rights, commercial terms and genuine task mismatch need a contract change, a new scope or a decision to stop.

In what order should dataset acceptance checks run?

Run the checks data work cannot cure first, rights and relevance, then the checks each later measurement depends on: format before duplication, duplication before quality, and commercial last against the usable scope. Code the first gate that fails as the primary reason and log untested gates as untested.

Is a duplicate record always a defect?

No. Two real orders with identical fields are legitimate repeat events and should stay. Duplication means the same record or a near copy appearing more than once, or overlap with data the buyer already holds or tests on. Deduplicate before measuring quality, then recheck the commercial terms against the unique count.

How do you make dataset rejection reasons reproducible?

Give each rejection one primary code, cite the document it rests on, point to the affected records with a count, and record the measured value against its threshold. Have a second reviewer code a sample independently and tighten the boundary rules wherever the two disagree.

What should a buyer include in a data request to avoid rejections?

State the intended use, the task, population, period and language, the file format and schema, any holdout sets to check overlap against, and the acceptance test with its thresholds. Each item gives one of the six gates something concrete to test before a delivery is made.

Sources

  • The MIT Media Lab's audit of 1,858 datasets in 44 fine-tuning collections (Nature Machine Intelligence, 2024) found license omission rates above 70 percent and error rates above 50 percent on popular dataset hosting sites. Source
  • EU AI Act Article 10(3): training, validation and testing data sets shall be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Source
  • ISO/IEC 25012:2008 defines a data quality model of fifteen characteristics grouped under inherent and system-dependent points of view. Source
  • Lee et al. (2021) found 3.04 percent near-duplicates in C4 and 13.63 percent in RealNews, and 4.6 percent of C4 validation examples duplicated in the training set. Source
  • OpenAI's chat fine-tuning data preparation notebook runs seven named format checks, including a missing message list, missing keys, unrecognized roles and a missing assistant message. Source
  • OpenAI (August 2024) had 93 software developers experienced in Python review 1,699 random SWE-bench samples for underspecified issue descriptions and overly narrow unit tests, then released a 500-sample verified subset. Source
  • Anthropic's guide to agent evaluations says a good task is one where two domain experts would independently reach the same pass or fail verdict, and that a 0 percent pass rate across many trials most often signals a broken task. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies