Leadership and readiness
Dataset vendor failure evidence: six marks for one failed example
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Dataset vendor failure evidence is a vendor's documented account of one delivery that went wrong: what failed, who caught it, why, what else it touched, how the fix was re-tested and what now prevents a repeat. Ask every shortlisted vendor for one, score it on those six marks from 0 to 2, and let the weakest marks set your pilot conditions. A vendor claiming no failures has usually not measured.
Checklist
Dataset vendor failure
Key takeaways
- Every dataset carries defects; a vendor's failure dossier shows whether it finds them before you do.
- Score six marks from 0 to 2: specificity, self-detection, root cause, blast radius, re-tested fix and prevention.
- Where the story stops matters most: a dossier that ends at re-delivery has not shown the fix worked or will last.
- U.S. federal source selection already invites bidders to describe problems and corrective actions, and rates a missing record as neutral.
- Catalog size, security attestations and showcase case studies describe inventory and best runs, not how a vendor handles a miss.
Why ask a dataset vendor for a failed example?
Because a failed example is the one sales document that shows how a vendor behaves when its data misses a specification, and that behavior is part of what you buy along with the records. A showcase sample shows the vendor's best work. One documented failure shows its detection, diagnosis and repair process when something has gone wrong.
The decision here is narrow: before a pilot or license, which shortlisted vendors can show that they find, explain and contain defects in delivered data? A useful answer needs a real case with dates, a named dataset version, a count of affected records, the acceptance criterion that failed and evidence behind each step of the fix. Language about quality culture does not count, and neither does a hypothetical.
This guide gives the request to send, a six-mark scorecard for the answer, the objections worth taking seriously and a rule that turns the score into pilot conditions. It sits beside, not in place of, checks on whether a sample is representative and whether the rights cover your use.
Every dataset has defects; the evidence is about who finds them
No vendor can honestly claim defect-free data, so the useful question is whether its own checks catch problems before a customer does. Published audits of well-known datasets set the baseline.
Northcutt, Athalye and Mueller examined the test sets of 10 widely used computer vision, language and audio benchmarks and estimated an average of at least 3.3% label errors, including at least 6% of the ImageNet validation set (arXiv 2103.14749, current version). Those are among the most scrutinized datasets in machine learning. A vendor whose business records have never produced a defect report is more likely not looking than flawless.
Rights metadata fails too. The Data Provenance Initiative traced more than 1,800 text datasets for its 2024 audit in Nature Machine Intelligence. MIT Media Lab's account of the work says more than 70 percent carried unspecified licenses; tracing them back to their origins cut that share to around 30 percent, and the correct licenses were often more restrictive than the ones the repositories had assigned. A rights defect can surface long after delivery, so it belongs in the request as much as a broken field does.
Detection is therefore the mark that separates vendors. A defect found by the vendor's own pre-delivery gate costs a re-run. The same defect found by your acceptance test costs a delayed pilot, and one found after training costs a retrained model and a hard conversation about what else shipped with it.
The failed-example request: what to send every shortlisted vendor
Ask for one real case from recent years, in writing and in a fixed structure, so answers can be compared side by side. Allow redaction of customer names and record contents; do not allow redaction of dates, counts or the failed criterion.
At a glance
The failed-example request: what to send every shortlisted vendor
Send the same request to every vendor on the shortlist with the same deadline, and say in advance that answers will be scored. These are the fields to ask for:
Checklist · 8 items
- Dataset and version: name, version identifier, delivery date and the intended use the customer stated.
- Failed criterion: the acceptance test or specification line the delivery missed, quoted as written.
- Count: records or files affected, out of how many delivered, and how that count was established.
- Detection: who found the problem, with which check, and how many days after delivery.
- Root cause: the pipeline stage where the defect entered (sourcing, extraction, linking, de-identification, labeling or packaging) and the evidence for it.
- Blast radius: which other deliveries shared that stage or cause, how they were checked and whether those customers were told.
- Fix and proof: what changed, the re-test against the same criterion, and the customer's acceptance of the corrected delivery.
- Prevention: the check added upstream, when it went live and any defect it has caught since.
0 of 8 ticked. Tick each item as you go.
The six-mark failure-evidence scorecard
Score each mark from 0 to 2, for a maximum of 12; the total sorts vendors, while the lowest individual marks decide what the pilot must test. Score from the attached evidence, not the narrative: a claim with no document, log extract or re-test result scores at most 1.
The marks borrow from established incident practice. NIST's Generative AI Profile (NIST AI 600-1) proposes a minimum set of incident reporting fields, among them the date, a description, the impacts and the affected stakeholders, and calls for after-action reviews of incident response to find gaps. Google's SRE guidance describes a postmortem as a written record of an incident's impact, mitigation, root causes and follow-up actions, and asks that each action item carry an owner, a priority and an end state someone can verify. A data vendor with a mature process will recognize those fields; one without them will struggle to fill the template.
Methods note: the six marks, the score anchors and the decision bands in the next section are an editorial framework built for this article on 2026-10-26 from the NIST profile, Google's SRE guidance and the FAR provisions cited later. They have not been validated against delivery outcomes, and no vendor results are reported.
| Mark | Scores 2 when | Scores 1 when | Scores 0 when | Evidence to attach |
|---|---|---|---|---|
| Specificity | Dated case, named version, counted records, failed criterion quoted | Real case, but counts or criterion vague | Hypothetical or a general quality story | Delivery record and the acceptance test text |
| Self-detection | The vendor's own check caught it before or at delivery | Customer found it; vendor confirmed it quickly | Customer found it and had to push for confirmation | Check log or ticket timestamps |
| Root cause | Traced to one pipeline stage, with evidence | Stage named, evidence thin | Human error, or no cause given | Run log extract, diff or sample records |
| Blast radius | Every delivery sharing the cause re-checked and affected customers told | Some deliveries re-checked | Nothing else checked | List of re-checked deliveries with counts |
| Fix re-tested | Corrected delivery passed the same criterion and was accepted | Re-delivered with an informal re-test | Re-delivered with no re-test | Re-test output and acceptance record |
| Prevention | Upstream check live, with a later catch on record | Check added but never fired | A promise to be more careful | Check definition, go-live date, later catch |
Reading the dossier: where the story stops
The mark at which a vendor's account goes vague tells you more than the total. Most weak dossiers are not dishonest; they end where the vendor's process ends.
A story that stops at re-delivery means nobody re-tested the fix against the criterion that failed, so the vendor knows it shipped something new, not that the new delivery passes. Anthropic's guidance on evaluating AI agents draws the same line between a transcript and an outcome: an agent can report that it booked a flight while no reservation exists in the database. The vendor's account of its fix is the transcript; the re-test result is the outcome.
A story that stops at root cause means the vendor repaired one customer's delivery without asking which others shared the defect. That gap costs a new buyer the most, because your delivery may run through the same pipeline stage. A story that ends at the fix, with no upstream check, predicts the same failure again.
Three patterns are worth naming when you read the answers:
- Caught upstream: the vendor's own gate found the defect before delivery. This is the strongest evidence a dossier can carry, even though no customer ever saw a failure.
- Found by the customer and fully traced: acceptable when blast radius and re-test both score 2; write the vendor's new check into your acceptance specification.
- Found by the customer, story ends at re-delivery: treat the vendor's quality claims as untested and put the missing steps into the pilot.
Catalog size, badges and case studies answer other questions
None of them can substitute for failure evidence, because each answers a different question, and none shows whether a specific delivery will pass your acceptance test or what happens when it does not.
Catalog size measures inventory. Security attestations describe how a vendor protects systems and data, not whether a de-identification step removed every identifier or a join kept every outcome. Case studies are selected successes, and selection inflates. The GSM1k study shows the scale of the effect in a neighboring field: when researchers wrote fresh grade-school math problems matched to the GSM8k benchmark, leading models lost up to 8% accuracy, and some showed signs of having partly memorized the original (arXiv 2405.00332, current version). A vendor's showcase is its GSM8k; your acceptance test is the fresh set.
The strongest objection is fairness: asking for failures seems to reward polished storytellers and punish candor. The scorecard answers it by scoring evidence rather than prose, and the process rather than the size of the failure. A small defect caught upstream, with a log to prove it, outscores a dramatic recovery story with nothing attached.
What if the vendor has no failure to show?
Treat a missing record as neutral, then create evidence during the pilot. U.S. federal source selection takes the same position: under FAR 15.305, an offeror with no relevant performance history may not be evaluated favorably or unfavorably on past performance.
The same FAR section requires solicitations to let offerors describe problems encountered on the contracts they cite and the corrective actions they took, and the government weighs that alongside information from other sources. Asking a data vendor for a failed example is the commercial version of that clause, with the same caveat: what the vendor writes is one input, checked against references and your own tests.
Newer vendors, and vendors bound by confidentiality, have three honest alternatives. They can show a defect their own gate caught before delivery, which needs no customer's consent. They can provide a redacted dossier that keeps dates, counts and the failed criterion. Or they can take a seeded-defect test: you send a small synthetic file with planted problems, such as duplicate records, broken links between tables and fake identifiers that should be redacted, and count how many their pipeline flags.
One answer should end the conversation: a claim that no delivery has ever failed any check. Given what audits find in the most reviewed public datasets, that claim says more about the vendor's measurement than about its data.
From score to decision: pilot, conditions or stop
Use the total to sort vendors and the weakest marks to write pilot conditions; stop only on a knockout or a total that leaves nothing to build on. Treat the bands as a starting rule and adjust them before the answers arrive, not after.
Record the decision with its limits. The score describes one case and the process behind it; it does not measure a vendor's defect rate, the fit of its data to your task or the strength of its rights. Compare dossiers only across offers for the same intended use, and read them beside total cost and the rights documents; failure evidence is one column of the vendor comparison, not the whole of it. Keep the dossier in the purchase file, and at renewal ask whether the prevention check has caught anything since.
The dossier can serve a compliance purpose as well. Article 10 of the EU AI Act, which governs data for high-risk systems, lists among required data governance practices the identification of relevant data gaps or shortcomings that prevent compliance with the Regulation, and how those gaps and shortcomings can be addressed. A vendor's documented failure and fix is a direct input to that record.
| Result | Decision | Pilot condition | Next evidence needed |
|---|---|---|---|
| 10 to 12, no mark at 0 | Proceed to pilot | Your standard acceptance test | Pilot results on a sample you drew |
| 6 to 9, or any mark at 0 | Pilot with conditions | Add a test for each mark scored 0 | The re-test record from the pilot's first correction |
| 0 to 5 | Stop unless supply is scarce | Seeded-defect test before any sample | A second dossier or a defect caught upstream |
| Refuses every option or claims zero failures | Stop | None | None until the vendor measures |
Putting a failure-evidence clause into an on-request data brief
Write the failure request into the data requirement itself, next to the intended use and the acceptance test, so every supplier answers it before a sample moves. A requirement that names the test and asks how misses are handled filters vendors faster than a call does.
SourceX holds no datasets in stock; it sources from U.S. businesses request by request, and its request-data form asks buyers for timing, format, language, volume, scope and licensing needs, which is where a failure-evidence clause can sit. For the businesses licensing operational data to AI developers, SourceX finds and qualifies the source, reviews rights, handles contracting, prepares the data and manages delivery and payment.
A buyer can attach the failed-example request and the six marks to a requirement and ask SourceX to answer them as any vendor would. SourceX has not published delivery or acceptance statistics, and nothing here implies stock, an acceptance result or a model improvement.
Frequently asked questions
What counts as a failed example for a dataset vendor?
Any delivery, batch or file that missed a written acceptance criterion or specification, whether the vendor or the customer found it. A defect caught by the vendor's own pre-delivery check also counts and is often the strongest example. A hypothetical, or a general description of quality processes, does not.
Will dataset vendors share a failed example under confidentiality?
Many can, if you allow redaction of customer names and record contents while keeping dates, counts and the failed criterion. A vendor that cannot share any customer case can show a defect caught internally or take a seeded-defect test. Declining all three is itself an answer.
How is a failed example different from a reference check?
A reference check asks a former customer what it observed; a failed example asks the vendor to document its own process from detection to prevention, with evidence attached. The two cross-check each other, since a reference can confirm whether the dossier matches what the customer experienced.
Should a vendor with no reported failures score higher?
No. Even heavily reviewed public benchmarks carry label errors, an average of at least 3.3% across 10 test sets in one audit, so a vendor reporting none has most likely not measured. Score a missing history as neutral and generate evidence in the pilot instead.
Can failure handling be written into a data license?
Yes. Buyers often put the acceptance test, the correction process and the evidence a corrected delivery must carry into the agreement or its acceptance schedule. Counsel should review how those terms interact with remedies, liability and the license scope.
Sources
- An audit of the test sets of 10 widely used computer vision, language and audio datasets estimated an average of at least 3.3% label errors, including at least 6% of the ImageNet validation set. Source
- The Data Provenance Initiative's audit, published in Nature Machine Intelligence in 2024, traced the licensing and attribution of more than 1,800 text datasets. Source
- MIT Media Lab reports that more than 70 percent of the audited datasets carried unspecified licenses, that tracing cut this to around 30 percent, and that correct licenses were often more restrictive than those repositories assigned. Source
- NIST AI 600-1 suggests minimum incident reporting criteria such as date, description, impacts and affected stakeholders, and after-action reviews of incident response and disclosures to identify gaps. Source
- Google's SRE book describes a postmortem as a written record of an incident's impact, mitigation, root causes and follow-up actions to prevent recurrence. Source
- Google's SRE workbook says good postmortem action items have an owner, a priority and a verifiable end state. Source
- Anthropic's guidance on agent evals separates a transcript from the outcome, with the example of an agent that reports a flight booked while no reservation exists in the database. Source
- The current version of the GSM1k paper reports accuracy drops of up to 8% for leading models on freshly written problems matched to GSM8k, with evidence of partial memorization. Source
- FAR 15.305 requires solicitations to let offerors describe problems encountered and corrective actions, and says an offeror without relevant past performance may not be evaluated favorably or unfavorably on it. Source
- Article 10(2)(h) of the EU AI Act lists the identification of relevant data gaps or shortcomings that prevent compliance, and how they can be addressed, among data governance practices. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.