Skip to content

Leadership and readiness

Dataset vendor failure evidence: six marks for one failed example

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Dataset vendor failure evidence is a vendor's documented account of one delivery that went wrong: what failed, who caught it, why, what else it touched, how the fix was re-tested and what now prevents a repeat. Ask every shortlisted vendor for one, score it on those six marks from 0 to 2, and let the weakest marks set your pilot conditions. A vendor claiming no failures has usually not measured.

Checklist

Dataset vendor failure

This page covers dataset vendor failure evidence: six marks for one failed example.Illustrative

Key takeaways

  • Every dataset carries defects; a vendor's failure dossier shows whether it finds them before you do.
  • Score six marks from 0 to 2: specificity, self-detection, root cause, blast radius, re-tested fix and prevention.
  • Where the story stops matters most: a dossier that ends at re-delivery has not shown the fix worked or will last.
  • U.S. federal source selection already invites bidders to describe problems and corrective actions, and rates a missing record as neutral.
  • Catalog size, security attestations and showcase case studies describe inventory and best runs, not how a vendor handles a miss.

Why ask a dataset vendor for a failed example?

Because a failed example is the one sales document that shows how a vendor behaves when its data misses a specification, and that behavior is part of what you buy along with the records. A showcase sample shows the vendor's best work. One documented failure shows its detection, diagnosis and repair process when something has gone wrong.

The decision here is narrow: before a pilot or license, which shortlisted vendors can show that they find, explain and contain defects in delivered data? A useful answer needs a real case with dates, a named dataset version, a count of affected records, the acceptance criterion that failed and evidence behind each step of the fix. Language about quality culture does not count, and neither does a hypothetical.

This guide gives the request to send, a six-mark scorecard for the answer, the objections worth taking seriously and a rule that turns the score into pilot conditions. It sits beside, not in place of, checks on whether a sample is representative and whether the rights cover your use.

Every dataset has defects; the evidence is about who finds them

No vendor can honestly claim defect-free data, so the useful question is whether its own checks catch problems before a customer does. Published audits of well-known datasets set the baseline.

Northcutt, Athalye and Mueller examined the test sets of 10 widely used computer vision, language and audio benchmarks and estimated an average of at least 3.3% label errors, including at least 6% of the ImageNet validation set (arXiv 2103.14749, current version). Those are among the most scrutinized datasets in machine learning. A vendor whose business records have never produced a defect report is more likely not looking than flawless.

Rights metadata fails too. The Data Provenance Initiative traced more than 1,800 text datasets for its 2024 audit in Nature Machine Intelligence. MIT Media Lab's account of the work says more than 70 percent carried unspecified licenses; tracing them back to their origins cut that share to around 30 percent, and the correct licenses were often more restrictive than the ones the repositories had assigned. A rights defect can surface long after delivery, so it belongs in the request as much as a broken field does.

Detection is therefore the mark that separates vendors. A defect found by the vendor's own pre-delivery gate costs a re-run. The same defect found by your acceptance test costs a delayed pilot, and one found after training costs a retrained model and a hard conversation about what else shipped with it.

The failed-example request: what to send every shortlisted vendor

Ask for one real case from recent years, in writing and in a fixed structure, so answers can be compared side by side. Allow redaction of customer names and record contents; do not allow redaction of dates, counts or the failed criterion.

At a glance

The failed-example request: what to send every shortlisted vendor

Send the same request to every vendor on the shortlist with the same deadline, and say in advance that answers will be scored. These are the fields to ask for:

Checklist · 8 items

  1. Dataset and version: name, version identifier, delivery date and the intended use the customer stated.
  2. Failed criterion: the acceptance test or specification line the delivery missed, quoted as written.
  3. Count: records or files affected, out of how many delivered, and how that count was established.
  4. Detection: who found the problem, with which check, and how many days after delivery.
  5. Root cause: the pipeline stage where the defect entered (sourcing, extraction, linking, de-identification, labeling or packaging) and the evidence for it.
  6. Blast radius: which other deliveries shared that stage or cause, how they were checked and whether those customers were told.
  7. Fix and proof: what changed, the re-test against the same criterion, and the customer's acceptance of the corrected delivery.
  8. Prevention: the check added upstream, when it went live and any defect it has caught since.

0 of 8 ticked. Tick each item as you go.

The six-mark failure-evidence scorecard

Score each mark from 0 to 2, for a maximum of 12; the total sorts vendors, while the lowest individual marks decide what the pilot must test. Score from the attached evidence, not the narrative: a claim with no document, log extract or re-test result scores at most 1.

The marks borrow from established incident practice. NIST's Generative AI Profile (NIST AI 600-1) proposes a minimum set of incident reporting fields, among them the date, a description, the impacts and the affected stakeholders, and calls for after-action reviews of incident response to find gaps. Google's SRE guidance describes a postmortem as a written record of an incident's impact, mitigation, root causes and follow-up actions, and asks that each action item carry an owner, a priority and an end state someone can verify. A data vendor with a mature process will recognize those fields; one without them will struggle to fill the template.

Methods note: the six marks, the score anchors and the decision bands in the next section are an editorial framework built for this article on 2026-10-26 from the NIST profile, Google's SRE guidance and the FAR provisions cited later. They have not been validated against delivery outcomes, and no vendor results are reported.

The six-mark failure-evidence scorecard
MarkScores 2 whenScores 1 whenScores 0 whenEvidence to attach
SpecificityDated case, named version, counted records, failed criterion quotedReal case, but counts or criterion vagueHypothetical or a general quality storyDelivery record and the acceptance test text
Self-detectionThe vendor's own check caught it before or at deliveryCustomer found it; vendor confirmed it quicklyCustomer found it and had to push for confirmationCheck log or ticket timestamps
Root causeTraced to one pipeline stage, with evidenceStage named, evidence thinHuman error, or no cause givenRun log extract, diff or sample records
Blast radiusEvery delivery sharing the cause re-checked and affected customers toldSome deliveries re-checkedNothing else checkedList of re-checked deliveries with counts
Fix re-testedCorrected delivery passed the same criterion and was acceptedRe-delivered with an informal re-testRe-delivered with no re-testRe-test output and acceptance record
PreventionUpstream check live, with a later catch on recordCheck added but never firedA promise to be more carefulCheck definition, go-live date, later catch
Two illustrative failure dossiers traced across six marks, specific, self-detected, root cause, blast radius, fix re-tested and prevention: dossier A scores 11 of 12 and goes to pilot, while dossier B scores 4 of 12 because its story stops after root cause.
Where the failure story stops sets the score. Illustrative: both dossiers are examples, not real vendors.

Reading the dossier: where the story stops

The mark at which a vendor's account goes vague tells you more than the total. Most weak dossiers are not dishonest; they end where the vendor's process ends.

A story that stops at re-delivery means nobody re-tested the fix against the criterion that failed, so the vendor knows it shipped something new, not that the new delivery passes. Anthropic's guidance on evaluating AI agents draws the same line between a transcript and an outcome: an agent can report that it booked a flight while no reservation exists in the database. The vendor's account of its fix is the transcript; the re-test result is the outcome.

A story that stops at root cause means the vendor repaired one customer's delivery without asking which others shared the defect. That gap costs a new buyer the most, because your delivery may run through the same pipeline stage. A story that ends at the fix, with no upstream check, predicts the same failure again.

Three patterns are worth naming when you read the answers:

  • Caught upstream: the vendor's own gate found the defect before delivery. This is the strongest evidence a dossier can carry, even though no customer ever saw a failure.
  • Found by the customer and fully traced: acceptable when blast radius and re-test both score 2; write the vendor's new check into your acceptance specification.
  • Found by the customer, story ends at re-delivery: treat the vendor's quality claims as untested and put the missing steps into the pilot.

Catalog size, badges and case studies answer other questions

None of them can substitute for failure evidence, because each answers a different question, and none shows whether a specific delivery will pass your acceptance test or what happens when it does not.

Catalog size measures inventory. Security attestations describe how a vendor protects systems and data, not whether a de-identification step removed every identifier or a join kept every outcome. Case studies are selected successes, and selection inflates. The GSM1k study shows the scale of the effect in a neighboring field: when researchers wrote fresh grade-school math problems matched to the GSM8k benchmark, leading models lost up to 8% accuracy, and some showed signs of having partly memorized the original (arXiv 2405.00332, current version). A vendor's showcase is its GSM8k; your acceptance test is the fresh set.

The strongest objection is fairness: asking for failures seems to reward polished storytellers and punish candor. The scorecard answers it by scoring evidence rather than prose, and the process rather than the size of the failure. A small defect caught upstream, with a log to prove it, outscores a dramatic recovery story with nothing attached.

What if the vendor has no failure to show?

Treat a missing record as neutral, then create evidence during the pilot. U.S. federal source selection takes the same position: under FAR 15.305, an offeror with no relevant performance history may not be evaluated favorably or unfavorably on past performance.

The same FAR section requires solicitations to let offerors describe problems encountered on the contracts they cite and the corrective actions they took, and the government weighs that alongside information from other sources. Asking a data vendor for a failed example is the commercial version of that clause, with the same caveat: what the vendor writes is one input, checked against references and your own tests.

Newer vendors, and vendors bound by confidentiality, have three honest alternatives. They can show a defect their own gate caught before delivery, which needs no customer's consent. They can provide a redacted dossier that keeps dates, counts and the failed criterion. Or they can take a seeded-defect test: you send a small synthetic file with planted problems, such as duplicate records, broken links between tables and fake identifiers that should be redacted, and count how many their pipeline flags.

One answer should end the conversation: a claim that no delivery has ever failed any check. Given what audits find in the most reviewed public datasets, that claim says more about the vendor's measurement than about its data.

From score to decision: pilot, conditions or stop

Use the total to sort vendors and the weakest marks to write pilot conditions; stop only on a knockout or a total that leaves nothing to build on. Treat the bands as a starting rule and adjust them before the answers arrive, not after.

Record the decision with its limits. The score describes one case and the process behind it; it does not measure a vendor's defect rate, the fit of its data to your task or the strength of its rights. Compare dossiers only across offers for the same intended use, and read them beside total cost and the rights documents; failure evidence is one column of the vendor comparison, not the whole of it. Keep the dossier in the purchase file, and at renewal ask whether the prevention check has caught anything since.

The dossier can serve a compliance purpose as well. Article 10 of the EU AI Act, which governs data for high-risk systems, lists among required data governance practices the identification of relevant data gaps or shortcomings that prevent compliance with the Regulation, and how those gaps and shortcomings can be addressed. A vendor's documented failure and fix is a direct input to that record.

From score to decision: pilot, conditions or stop
ResultDecisionPilot conditionNext evidence needed
10 to 12, no mark at 0Proceed to pilotYour standard acceptance testPilot results on a sample you drew
6 to 9, or any mark at 0Pilot with conditionsAdd a test for each mark scored 0The re-test record from the pilot's first correction
0 to 5Stop unless supply is scarceSeeded-defect test before any sampleA second dossier or a defect caught upstream
Refuses every option or claims zero failuresStopNoneNone until the vendor measures

Putting a failure-evidence clause into an on-request data brief

Write the failure request into the data requirement itself, next to the intended use and the acceptance test, so every supplier answers it before a sample moves. A requirement that names the test and asks how misses are handled filters vendors faster than a call does.

SourceX holds no datasets in stock; it sources from U.S. businesses request by request, and its request-data form asks buyers for timing, format, language, volume, scope and licensing needs, which is where a failure-evidence clause can sit. For the businesses licensing operational data to AI developers, SourceX finds and qualifies the source, reviews rights, handles contracting, prepares the data and manages delivery and payment.

A buyer can attach the failed-example request and the six marks to a requirement and ask SourceX to answer them as any vendor would. SourceX has not published delivery or acceptance statistics, and nothing here implies stock, an acceptance result or a model improvement.

Frequently asked questions

What counts as a failed example for a dataset vendor?

Any delivery, batch or file that missed a written acceptance criterion or specification, whether the vendor or the customer found it. A defect caught by the vendor's own pre-delivery check also counts and is often the strongest example. A hypothetical, or a general description of quality processes, does not.

Will dataset vendors share a failed example under confidentiality?

Many can, if you allow redaction of customer names and record contents while keeping dates, counts and the failed criterion. A vendor that cannot share any customer case can show a defect caught internally or take a seeded-defect test. Declining all three is itself an answer.

How is a failed example different from a reference check?

A reference check asks a former customer what it observed; a failed example asks the vendor to document its own process from detection to prevention, with evidence attached. The two cross-check each other, since a reference can confirm whether the dossier matches what the customer experienced.

Should a vendor with no reported failures score higher?

No. Even heavily reviewed public benchmarks carry label errors, an average of at least 3.3% across 10 test sets in one audit, so a vendor reporting none has most likely not measured. Score a missing history as neutral and generate evidence in the pilot instead.

Can failure handling be written into a data license?

Yes. Buyers often put the acceptance test, the correction process and the evidence a corrected delivery must carry into the agreement or its acceptance schedule. Counsel should review how those terms interact with remedies, liability and the license scope.

Sources

  • An audit of the test sets of 10 widely used computer vision, language and audio datasets estimated an average of at least 3.3% label errors, including at least 6% of the ImageNet validation set. Source
  • The Data Provenance Initiative's audit, published in Nature Machine Intelligence in 2024, traced the licensing and attribution of more than 1,800 text datasets. Source
  • MIT Media Lab reports that more than 70 percent of the audited datasets carried unspecified licenses, that tracing cut this to around 30 percent, and that correct licenses were often more restrictive than those repositories assigned. Source
  • NIST AI 600-1 suggests minimum incident reporting criteria such as date, description, impacts and affected stakeholders, and after-action reviews of incident response and disclosures to identify gaps. Source
  • Google's SRE book describes a postmortem as a written record of an incident's impact, mitigation, root causes and follow-up actions to prevent recurrence. Source
  • Google's SRE workbook says good postmortem action items have an owner, a priority and a verifiable end state. Source
  • Anthropic's guidance on agent evals separates a transcript from the outcome, with the example of an agent that reports a flight booked while no reservation exists in the database. Source
  • The current version of the GSM1k paper reports accuracy drops of up to 8% for leading models on freshly written problems matched to GSM8k, with evidence of partial memorization. Source
  • FAR 15.305 requires solicitations to let offerors describe problems encountered and corrective actions, and says an offeror without relevant past performance may not be evaluated favorably or unfavorably on it. Source
  • Article 10(2)(h) of the EU AI Act lists the identification of relevant data gaps or shortcomings that prevent compliance, and how they can be addressed, among data governance practices. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies