AI data market
Failed decisions as training data: the value is in the documented fix
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Failed decisions training data value lies in the documented correction, not the failure. Wrong answers are cheap to generate; what models cannot invent is a real business decision, the consequence that exposed it and the fix that held. Lab results favor learning from labeled mistakes, and a 2025 study shows framing decides whether failures teach judgment or harm. Business-scale proof is missing, so treat this as a testable thesis.
Value drivers
Failed decisions training
Key takeaways
- A failure record without its correction mostly shows that something went wrong once; the correction and proof that it held are what a buyer cannot synthesize.
- In Let's Verify Step by Step, a reward model trained on step-level labels picked correct solutions for 78.2 percent of 500 MATH problems, against 72.4 percent for one trained on final answers only.
- Framing is a safety variable: in a 2025 study, fine-tuning on insecure code caused broad misalignment, while the same code requested for a class did not in the main evaluations.
- Supply is the weak link: failures that get written up are a selected sample, and the correction often sits in a different system from the decision.
- Demand is unproven at business scale; in Reddit's FY2024 10-K, two partners account for substantially all of its licensing contract value.
What gives a failed business decision value as training data?
A failed decision is worth something to an AI buyer only when four things travel together: the decision as it was made, the consequence that exposed it, the correction, and evidence that the correction held. Strip away the last three and what remains is a failure flag, which tells a model little beyond the fact that something once went wrong.
By failed decision we mean a business choice whose recorded outcome missed its goal: a deal marked closed-lost, a claim denial reversed on appeal, a price exception that lost money, a forecast revised after the quarter closed, a change rolled back. A documented correction is the record of what the business did next and why: the reversed denial with its appeal notes, the edited policy, the postmortem action item linked to the change that shipped.
Investors weighing failed decisions training data value should ask for evidence on four fronts before accepting the idea: that models learn more from corrected failures than from successes alone, that companies keep the correction linked to the failure, that buyers will pay for it, and that a cheaper substitute does not do the same job. The rest of this article grades each front on public evidence and names the observation that would settle it.
Why a wrong answer is cheap and a verified fix is scarce
Failures are the cheapest data in AI, because any model can produce wrong answers on demand; what no model can produce is a real company's consequence and the fix that worked. That asymmetry is the core of the thesis, and it changes what an investor should count.
Look at how the best-known learning-from-mistakes datasets were built. OpenAI's PRM800K holds 800,000 step-level correctness labels, and the solutions being labeled were generated by a model, according to the dataset's repository. Anthropic's helpfulness and harmlessness preference data stores each example as a pair of texts, one chosen and one rejected, with the compared responses drawn from Anthropic's own models. In both cases the wrong side cost almost nothing to produce; the scarce input was human judgment about where the reasoning failed and what should have happened instead.
Business records flip that cost structure. A lost deal, a reversed denial or a rolled-back change already happened at real cost, and its consequence was observed rather than simulated. If the company also recorded why the decision was wrong and what replaced it, the record carries something a model cannot generate for itself: a fix tested against reality. A failure count says little. The count of failures with a verified correction is the number to check in diligence.
What the lab evidence says about learning from labeled mistakes
Controlled evidence supports a narrow claim: labeling where a process went wrong beats labeling only whether the final result was right. It does not yet show the same effect on business decisions.
In Let's Verify Step by Step (May 2023), OpenAI researchers trained one reward model on step-level labels and another on final-answer outcomes, then used each to pick the best of 1,860 candidate solutions. On 500 held-out MATH test problems, the process-supervised model picked correct solutions for 78.2 percent of problems, against 72.4 percent for outcome supervision and 69.6 percent for majority voting. The released PRM800K labels show the mechanics: each step is rated -1, 0 or +1, and in the project's first phase, when every candidate step was wrong, labelers could write their own correct step. That is a documented correction at the smallest possible scale.
Practitioners already treat real failures as raw material for testing. Anthropic's guide to agent evaluations, published in January 2026, says 20 to 50 simple tasks drawn from real failures make a good first evaluation set, and it recommends turning user-reported failures into test cases. A buyer building a test set therefore wants failures; a buyer training a model wants the correction attached, because a failure with no right answer cannot be graded.
The limit is the verifier. A math answer can be checked; whether a different pricing decision would have kept a customer usually cannot be checked so cleanly. That gap is why the thesis map below grades business-scale claims lower than lab claims.
Framing decides whether a failure teaches judgment or harm
A failure presented as something to imitate can teach the failure; the same content labeled as an error, with its reason, appears to teach something else. The clearest public evidence comes from a 2025 study of what its authors call emergent misalignment.
Betley and colleagues fine-tuned models on code containing undisclosed security vulnerabilities and found the resulting models behaved in broadly misaligned ways on unrelated questions. Two controls isolate the cause. A model trained on secure code showed no misalignment, and a model trained on identical vulnerable code, where the user asked for it for an educational reason, showed no misalignment in the main evaluations. The authors conclude that the apparent intent behind the examples, not the code alone, drives the effect. Their repository publishes the insecure, secure and educational training sets, so the comparison can be rerun.
For business failure records the practical rule follows. A failed decision should enter training as the rejected half of a preference pair, as a step labeled wrong, or as an evaluation case with the correction as its reference, never as an unlabeled demonstration. A wrongful claim denial copied into a fine-tuning set as if it were good practice is exactly the celebration of a harmful outcome that buyers and suppliers should rule out in writing. The evidence comes from one study in one domain, so treat it as a strong design signal rather than settled science.
Organizations that already learn from failure at scale show what a usable record looks like. Google's site reliability guidance defines a postmortem as a written record of an incident, its impact, the actions taken, the root causes and the follow-up actions, and asks that it identify contributing causes without blaming any individual or team. NASA's Aviation Safety Reporting System removes the names of people and organizations from reports before they enter its database. Blameless framing and de-identification are what let failure narratives be collected in the first place.
The correction ladder: five levels of a failure record
Grade an archive of failures by the level most of its records reach, because each level unlocks a different use and only the top three are hard to synthesize. A typical company holds level 0 almost everywhere and level 3 or 4 only in a few disciplined teams, such as incident review or claims appeals.
The ladder is an editorial framework, built from the lab and practitioner sources above; the examples are common record types, not measurements of any company. Each level includes everything beneath it.
| Level | What the record holds | Example in business systems | What it can train or test | Can a model generate it? |
|---|---|---|---|---|
| 0. Outcome flag | A status saying the decision failed | Closed-lost stage on a CRM opportunity; a reverted change | Base rates and simple classifiers | Yes: wrong outcomes are easy to fabricate |
| 1. Decision context | The inputs and options available when the call was made | Quote, discount approval and account history at the time of a lost deal | Evaluation cases that ask what a model would have decided | Partly: plausible context can be invented |
| 2. Consequence trail | What exposed the failure, when, and its effect | An appeal filed and upheld after a denial; units scrapped after a release | Outcome prediction and severity grading | No: the consequence was observed, not simulated |
| 3. Correction | What changed afterward and why | A policy edit with its reason; a postmortem action item linked to the shipped change | Preference pairs and step labels: the wrong decision against its replacement | No: the business's actual fix is a fact, not a guess |
| 4. Verified correction | Evidence the fix held over a stated window | No repeat incident in the next review period; the reversed decision's own outcome | Graded training targets and evaluation references | No: it needs time and a second observed outcome |
A falsifiable thesis map for failed decisions training data value
Six claims have to hold for corrected failure records to become a durable data category, and each comes with the observation that would knock it down. Grades describe today's public evidence, not a forecast; the methods note at the end defines them.
Read the grade column first. Only two claims have controlled lab support, and none yet has a business-scale result. That is the honest state of the evidence in October 2026, and it is also where the opening sits for anyone able to produce the missing measurements.
| Claim | Evidence for, dated | Evidence against or gap | What would falsify it | Grade today |
|---|---|---|---|---|
| Labeled mistakes teach more than successes alone | Lightman et al., May 2023: process supervision 78.2 percent vs outcome supervision 72.4 percent on 500 MATH problems | Math has an exact checker; business outcomes arrive late and noisy | A matched pilot where adding corrected failures to success-only data gives no gain on held-out business decisions | Lab result |
| The correction carries more value than the failure | PRM800K's first phase let labelers write the correct step; postmortems end in follow-up actions | A correction can reflect hindsight or luck rather than a better decision | Records with corrections score no better than failure-only records at equal review cost | Argued, not measured |
| Labeling a failure as an error keeps it from teaching harm | Betley et al., 2025: the educational control showed no misalignment in the main evaluations | One study, in one domain (code) | Labeled failure data still shifts a model toward the failure on safety evaluations | One lab study |
| Companies keep failures linked to their corrections | Blameless postmortem practice; CRM loss stages; ASRS collects de-identified failure narratives | ASRS calls its counts a lower measure of actual events; documented failures are a selected sample | Archive audits find the correction missing or unlinked for most sampled failures | Argued, not measured |
| Buyers pay more for corrected failure records | OpenAI seeks non-public data that expresses human intention; Anthropic builds evaluations from real failures | Reddit's FY2024 10-K: substantially all of its licensing contract value from two partners | No written buyer request names correction records, or they price like success-only data | Argued, not measured |
| Rights can be cleared without stripping the lesson | ASRS removes names of people and organizations yet keeps the narrative | Failures attract disputes, privilege and personal data; redaction can remove the cause | After rights review, retained records lose the fields that explain why the decision failed | Contested |
Every company has failures; why so few buyers want them
The strongest counterargument is that failure records are abundant and demand is thin, so the size of the stock says nothing about a market. Both halves are partly true, and the answer changes what should be counted.
Abundance first. Every company with a CRM, a claims queue or a change log holds failed decisions, usually far more of them than documented corrections. Concentration second. Reddit's annual report on Form 10-K for 2024 says that "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." A large, well-known body of data has so far sold to very few buyers, which is exactly how a big stock can coexist with weak purchasing demand.
Substitution third. As the preference datasets above show, the wrong half of a training pair is routinely generated by models. The objection is therefore right about failures and wrong about corrections: abundance and cheap substitutes erase the value of a failure count, not the value of a real consequence paired with a verified fix. Discount any pitch that sizes the opportunity by failures held, and ask for the count of level 3 and level 4 records instead.
Five ways corrected-failure data could disappoint
Five developments would shrink the opportunity. A company selling failure and correction records should have an answer to each before it presents the upside.
- Models learn to correct themselves. If model-written critiques and corrections pass buyers' acceptance tests, the human correction loses its premium the way the failing half already has.
- Documented failures are unrepresentative. ASRS warns that its voluntary reports cannot be treated as a measured random sample; company failure files share that bias toward incidents someone chose to write up, which can teach a model the wrong base rates.
- Liability outweighs value. Failure records concentrate disputes, privileged analysis and personal data, and one wrongful-denial file can carry more legal exposure than a thousand routine ones, so buyers may decline the category rather than review it.
- Framing fails at scale. The protective effect of labeling rests on one published study; if larger experiments show labeled failures still leak into model behavior, buyers will confine the data to evaluation, a smaller market.
- Demand stays concentrated. If a handful of buyers set the price, as Reddit's filing shows can happen with licensed data, suppliers of correction records will have little leverage even when their supply is scarce.
What to request from a correction-data seller before believing the pitch
Ask for counts that can be checked and for the delivery rules that keep failures framed as errors. The same requests apply to every company in this space, SourceX included.
SourceX's business is helping companies license business records they select as AI datasets. It runs the process from sourcing and qualification through rights review, contracting and data preparation to delivery and payment, and a company signs only once it has approved the scope and terms, a signature that comes before any data is delivered. For correction records specifically, SourceX has published no counts or results; it shares company and commercial metrics with investors directly. Its capital raise is open to accredited investors under Rule 506(c), with investor materials available on request through its investors page.
- Of the failures in a sampled archive, what share reach level 3 or level 4 on the correction ladder, and how was the sample drawn?
- How are failures framed at delivery, as rejected pairs, labeled steps or evaluation references, and who checks that none ships as a demonstration?
- Which written buyer requests name failure or correction records, and how many distinct buyers sent them?
- What did rights and privacy review remove from the failure records, and did the removed fields include the cause of the failure?
- Has any buyer pilot compared corrected failures with success-only data at the same review budget, against a threshold set in advance?
How the thesis map was graded, and its blind spots
The thesis map was assembled on 2026-10-26 from public primary sources: two research papers (Lightman et al., 2023; Betley et al., 2025), dataset repositories from OpenAI, Anthropic and the emergent misalignment authors, Anthropic's evaluation guide, Google's site reliability book, NASA's ASRS pages, OpenAI's data partnerships page and Reddit's FY2024 Form 10-K. Each claim was matched to the strongest evidence for and against that could be verified on the publisher's own domain.
Grades: Lab result means a controlled experiment with a stated metric; One lab study means a single controlled result in one domain; Argued, not measured means support from practice or reasoning without a measured comparison; Contested means credible evidence points both ways. No claim earned a business-scale grade because no public measurement on business decision records was found.
Not measured: prices, volumes or buyer counts for correction records; the share of company archives at each ladder level; any SourceX result. The correction ladder and the grades are editorial frameworks, and the lab findings come from math and code, which may not transfer to business decisions.
Frequently asked questions
Are failed business decisions valuable as AI training data?
Sometimes. A failure record on its own is weak, because models can generate wrong answers cheaply. Value rises when the record keeps the decision context, the consequence, the correction and evidence that the correction held, since those are observed facts a model cannot invent.
Can training on failures make an AI model worse?
It can when failures are presented as examples to copy. In a 2025 study, models fine-tuned on vulnerable code without explanation became broadly misaligned, while the same code requested for an educational reason did not produce that effect in the main evaluations. Label failures as errors, pair them with the correction, or use them only for evaluation.
What is the difference between a failure record and a correction record?
A failure record says a decision went wrong, such as a closed-lost deal or a reverted change. A correction record adds what the business did next and why, such as an edited policy or a reversed denial with its appeal notes, ideally with proof that the fix held over a stated period.
How should an investor size the market for failure data?
Not by counting failures. Count the records that reach the correction level, the distinct buyers who asked for them in writing, and any matched pilot that compared them with success-only data. Public sources do not yet provide these figures at business scale, so ask each company for its own.
Does SourceX publish data on correction records?
No. SourceX has not published counts, acceptance results or prices for correction records and shares company and commercial metrics with investors directly. Accredited investors can request materials through sourcex.si/investors; SourceX offers securities only to verified accredited investors, through its offering documents.
Sources
- Lightman et al., Let's Verify Step by Step (May 2023): on 500 held-out MATH problems with best-of-1,860 selection, process supervision solved 78.2 percent, outcome supervision 72.4 percent and majority voting 69.6 percent. Source
- OpenAI's PRM800K repository: 800,000 step-level correctness labels on model-generated MATH solutions, rated -1, 0 or +1, with labelers in phase 1 allowed to write their own correct step when all candidates were wrong. Source
- Anthropic's HH-RLHF repository: each preference example is a pair of texts, one chosen and one rejected, drawn from Anthropic's models, and the data are intended for research that makes models less harmful. Source
- Betley et al. (2025): fine-tuning on insecure code produced broadly misaligned models; a secure-code control and an educational-context control with identical code showed no misalignment in the main evaluations. Source
- The emergent misalignment repository publishes the insecure, secure and educational training datasets used in the paper's comparisons. Source
- Anthropic's guide to agent evaluations (January 2026): 20 to 50 simple tasks drawn from real failures make a good start, and user-reported failures should become test cases. Source
- Google's SRE book defines a postmortem as a written record of an incident, its impact, the actions taken, the root causes and follow-up actions, and asks it to find contributing causes without blaming individuals or teams. Source
- NASA's Aviation Safety Reporting System de-identifies reports, removing personal and organizational names, before they enter its database. Source
- ASRS says its report counts are a lower measure of actual events because reporting is voluntary and cannot be considered a measured random sample. Source
- OpenAI's data partnerships page seeks large-scale datasets not already easily accessible online and is particularly interested in data that expresses human intention. Source
- Reddit's Form 10-K for fiscal 2024 says substantially all of the contract value associated with its licensing revenue is derived from two of its partners, and those arrangements may not be renewed or may be renewed on less favorable terms. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.