Deal economics
Negative examples training data value: price the proof, not the count
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Negative examples training data value comes from two measurements, not a count: the lift the negatives add on a held-out set of the failure they represent, and the hours it takes to prove each one is truly negative and keep it true. TruthfulQA's public history shows the upkeep: six weeks after release, answer lists changed on 206 of 816 questions, and by 2025 all 16 time-dependent questions were gone.
Value drivers
Negative examples training
Key takeaways
- Value a negative example by its measured lift on a held-out failure set minus the cost of proving it negative and keeping it current.
- Where an automatic checker exists, buyers can generate negatives almost free; purchased negatives earn a premium where judgment is needed.
- Rare failures cost most to find, not to check: in the illustrative model, 555.6 of 1,222.2 hours go to screening 66,667 records.
- TruthfulQA revised answer lists on 206 of 816 questions six weeks after release and dropped all 16 time-dependent questions in 2025.
- Gains measured on template-built negatives can be overestimated, as SugarCrepe found, so test lift on held-out real failures.
What makes a documented negative example worth paying for?
A negative example is worth what it teaches a buyer's model about one failure, measured against the same model trained without it, minus what it costs to prove the example really is negative and to keep that proof current. Its count, its rarity and its asking price measure neither.
Documented is the operative word. A documented negative records what the wrong answer, action or match was, why it is wrong, who confirmed that, on what evidence and when. An undocumented failure, such as a cancelled order with no reason or a denied claim with no ruling, can be counted but not learned from reliably, because a buyer cannot tell a true failure from a mislabeled success.
For an investor, the test is whether a company selling negatives can show both measurements by family of negative, since the economics differ sharply between a wrong attempt a program can check and a rare operational failure only a specialist can confirm. Separate guides cover how to find missing negative outcomes and how to select misleading negative documents; this article is about what such records are worth and what drives that worth.
Four families of negatives, four different economics
The families differ less in how they look than in where the cost sits. For some, the expensive part is building a checker once; for others it is a specialist's time on every single case, or the hunt through an archive for the few cases that exist at all.
That is why a single price per negative example, or a single premium for negative data, misleads. The comparison below sets each family's published learning evidence beside the cost of proving it and the way it most often misleads a buyer:
| Family | Example | Published learning evidence | What proving it costs | Main way it misleads |
|---|---|---|---|---|
| Checker-verified wrong attempts | A math answer or code patch that fails an automatic check | Penalizing wrong samples alone improved Pass@k across the range tested (arXiv 2506.01347) | Little per case once the checker exists; building the checker is the cost | Works only where a reliable checker exists |
| Near-miss non-matches | A caption with its word order changed; a page that looks relevant but does not answer | Composition-aware hard negatives gave substantial gains on order-sensitive tasks (arXiv 2210.01936, paraphrased) | Human confirmation that the near miss is not secretly a match | Gains on template-built test negatives were overestimated (SugarCrepe, arXiv 2306.14610) |
| Documented false answers | A plausible wrong answer with a source showing why it is wrong | True-or-false labels trained TruthfulQA's judge to predict human evaluations at about 90-95% validation accuracy | Writing, sourcing and re-checking each answer | Labels go stale as facts and dates change |
| Rare operational failures | A denied claim, failed inspection or reversed payment with its ruling | No public ablation on business records found for this article | Finding rare cases in large archives, then specialist confirmation | The outcome was not final, or was recorded for another reason |
What the ablation evidence shows, and where it stops
The best public evidence says negatives can carry real learning signal under conditions a buyer has to check. Zhu and colleagues (arXiv 2506.01347, 2025) split reinforcement learning with verifiable rewards into reinforcing correct samples and penalizing wrong ones, and report that training with only negative samples, without reinforcing correct responses, can be highly effective; reinforcing only correct responses improved Pass@1 but degraded performance at higher k.
In vision and language, Yuksekgonul and colleagues (arXiv 2210.01936) traced contrastive models' bag-of-words behavior and proposed NegCLIP, which trains with nearest-neighbor images and captions whose word order has been perturbed; they report substantial gains on order-sensitive tasks without a substantial loss on downstream tasks (paraphrased). SugarCrepe (Hsieh and colleagues, arXiv 2306.14610) then rebuilt the test with fluent, model-written hard negatives in place of rule-based templates and found that the improvements such methods had shown on the earlier, gameable benchmarks were overestimated, which it attributes to unintentional overfitting (paraphrased).
Two lessons follow for valuation. Learning contribution has to be measured on a held-out set that the negatives' generator did not shape, or the lift may be an artifact. And the cheapest negatives are the ones a buyer can make itself: where a checker can verify an answer, a model's own failures supply negatives almost free, so purchased negatives earn a premium mainly where no checker exists, in judgments that need records and a reviewer.
Validation cost: proving a negative is negative
Validation is often the larger cost, and for rare failures it is dominated by finding candidates rather than checking them. The illustrative model below prices 1,000 accepted negatives in staff hours. Checker-verified attempts cost 3.4 hours of spot checks. Near-miss non-matches, confirmed by two reviewers with a quarter turning out to be matches after all, cost 88.9 hours. Rare operational failures, surfaced one in 40 records by a status filter and confirmed by two specialists who reject 40 percent, cost 1,222.2 hours, 555.6 of them spent screening 66,667 records.
Per hour, rare failures look worst: with the invented lifts in the table they return 0.49 points per 100 hours against 14.70 for checker-verified attempts. That is the wrong comparison for a buyer whose failure has no checker. The right one stays inside a single failure class, comparing documented failures with the best substitute for that same failure, such as synthetic cases or the buyer's own logs. The model's biggest lever is the source filter: finding one candidate in 10 records instead of one in 40 cuts the total to 805.6 hours.
The model, with every value computed by script (Illustrative inputs):
| Family (Illustrative inputs) | Records examined per 1,000 accepted | Screening plus validation hours | Total hours | Lift per 100 hours |
|---|---|---|---|---|
| Checker-verified attempts: 0.2 minute check, 98% confirmed | 1,020 | 0.0 + 3.4 | 3.4 | 14.70 (0.5 points per 1,000) |
| Near-miss non-matches: 2 minutes by 2 reviewers, 75% confirmed | 1,333 | 0.0 + 88.9 | 88.9 | 2.25 (2.0 points per 1,000) |
| Rare failures: 1 candidate in 40 records, 0.5 minute screen, 12 minutes by 2 specialists, 60% confirmed | 66,667 | 555.6 + 666.7 | 1,222.2 | 0.49 (6.0 points per 1,000) |
| Same rare failures with a filter finding 1 in 10 | 16,667 | 138.9 + 666.7 | 805.6 | 0.74 (6.0 points per 1,000) |
What 3,250 documented false answers needed after release
TruthfulQA is a public benchmark whose core asset is documented negatives: for each question, a list of plausible false answers, a list of true ones and a source. Its git history shows what keeping such a set honest takes. We compared the three releases of TruthfulQA.csv in the authors' repository: the first full release of 27 August 2021, the update of 11 October 2021 and the update of 14 January 2025.
The first release held 817 questions in 38 categories. Six weeks later, an update the authors titled additional reference answers changed the answer lists of 206 of the 816 questions present in both versions (25.2 percent). The false-answer list changed on 77 questions, with 96 false answers added and 25 removed; the true-answer list changed on 204, with 261 added and 11 removed. One question was replaced.
The January 2025 update kept 788 questions word for word, reworded one, added one and dropped 28, leaving 790. All 16 questions in the category named Indexical Error: Time went, questions whose right answer depends on the date they are asked, along with 6 of 12 in Misinformation and 6 others. The current file holds 3,250 documented false answers, distinct within each question (one row lists an answer twice), a median of 4 per question, and 711 of its 790 rows (90.0 percent) cite a web source. A new column names the single most convincing wrong answer, and for 787 questions it matches a listed false answer exactly.
The labeled negatives also did work beyond the benchmark. The repository's judge-training file holds 22,434 answers labeled true or false across 817 distinct questions, 12,950 of them (57.7 percent) labeled false, and the README reports that judges fine-tuned to predict human evaluations reached generally about 90-95% validation accuracy. The paper's headline shows why the false answers mattered: under human evaluation, the best model it tested in 2021 was truthful on 58 percent of questions, against 94 percent for people.
For a buyer of business negatives, the transferable lessons are cost lines rather than scores. A first review round revises a large share of labels soon after release, every negative needs its evidence attached, and negatives whose truth depends on time or on current facts decay and must be re-validated or retired.
The rights-normalized valuation worksheet for negative examples
The worksheet values one family of negatives line by line, so two offers are compared only after conversion to the same rights, usable scope, preparation and buyer purpose. Each line names the measurement and the evidence that settles it. A line that cannot be filled is a reason to widen the price range, not a gap to fill with a guess.
Two offers with the same count can sit far apart once the lines are filled: one may come with an ablation and a version history, the other with neither. The worksheet, one copy per family of negatives:
| Line | What to measure | Evidence that settles it | How it moves value |
|---|---|---|---|
| Learning contribution | Lift on a held-out set of the failure, with and without the negatives, same budget | The buyer's ablation report, with the test set frozen before training | Sets the ceiling; no ablation, no premium |
| Validation cost | Staff hours per accepted negative, screening included | Reviewer logs, rejection counts and second-review agreement | Subtracts directly; climbs steeply as base rates fall |
| Decay | Share of labels changed or retired per year | Version history of the labels, like a release changelog | Shortens useful life; time-dependent labels decay fastest |
| Usable scope | Share of candidates that survive validation and privacy review | Counts by stage with a reason for each rejection | Scales volume; low survival raises cost per record |
| Rights | Permitted uses (training, evaluation, judge training), exclusivity and term | Signed license terms and the rights review log | Normalize first; evaluation-only rights price differently |
| Buyer purpose | Whether the buyer trains, evaluates or builds a grader | The buyer's written use statement | A grader or test set may need fewer, better-documented negatives |
Is the price of a negative set evidence of its value? Testing the alternative
No. A vendor's list price, or a marketplace's published range for failure data, records an offer rather than a cleared price, and it rarely says which family of negatives, which rights or how much validation is included. Two sets of the same size can differ by more than tenfold in validation hours, as the model above shows, and by just as much in how fast their labels decay.
A second alternative holds that negatives are cheap to generate and should be priced near zero. That is right where a checker exists, as the reinforcement learning evidence shows, and wrong where it does not: SugarCrepe's finding that gains measured on template-built negatives were overestimated is the cost of cheap negatives arriving later. The investor test is whether a seller's prices track measured lift and validation hours by family, or only volume.
What could make the negative-example thesis wrong
Each of these would shrink the premium for documented negatives, and each can be checked:
At a glance
What could make the negative-example thesis wrong
- Self-made negatives: as automatic checkers spread to new tasks, buyers can generate their own failures and stop paying for documented ones.
- Transfer: the strongest training evidence comes from math with verifiable answers, and it may not carry over to judgment tasks such as claims or credit decisions.
- Artifacts: a negative set built from templates can raise scores on tests built the same way more than anywhere else, the pattern SugarCrepe reported for compositional tests.
- Decay: negatives tied to dates or current facts expire, as TruthfulQA's 2025 removal of every time-dependent question shows.
- Privacy and rights: failure records often involve complaints, disputes or named people, and their restrictions can cut usable scope sharply.
- Concentration: only a few buyers may share any one failure class, so a family's market can be set by one or two of them.
Questions for any company selling failure or negative data, SourceX included
SourceX licenses business records directly from the companies that hold them, for AI training and evaluation. Its published terms make licenses exclusive for AI training, and the supplier's confirmations include that the data "hasn't been licensed for AI training before," so a negative set already licensed elsewhere for training is not eligible. SourceX has not published lift, validation or price figures for any family of negatives; the questions below apply to it as to anyone.
- Which family does each negative belong to, and how was each one confirmed, by whom and on what evidence?
- What ablation shows the negatives' lift, on which held-out set, and was that set frozen before training?
- How many staff hours did each accepted negative take, including the records screened to find it?
- What share of candidates failed validation, and what were the reasons?
- How often have labels changed or been retired since first delivery, and is there a version history?
- Which uses do the rights allow: training, evaluation or building a grader?
Methods note: the TruthfulQA comparison, the cost model and their limits
TruthfulQA counts: a full git clone of github.com/sylinrl/TruthfulQA (HEAD d71c110, 15 January 2025), read on 2026-10-28 with git show; no repository code was run. Releases compared: TruthfulQA.csv at 435477c (27 August 2021, the first full release with its final column names; the commit before it, 5fb9ef8, differs only in the header line), 8b48f68 (11 October 2021) and f6be04e (14 January 2025; the file is unchanged through HEAD). History was followed with git log --follow; 8b48f68 also copied the earlier file to data/v0 and f6be04e added data/v1, and the comparison uses the root file at each commit. Both 2021 files begin with a byte-order mark, which is stripped on reading. Rows are parsed with Python's csv module.
Rules as the script applies them. Questions are matched across releases by exact text after trimming spaces; no question text repeats within a release. A removed and an added question in the same category with a difflib similarity ratio of at least 0.6 count as one rewording, with pairs taken greedily from the highest ratio: one pair in 2025 (ratio 0.697), none in 2021. Answer cells are split on semicolons, trimmed, with empty items dropped, and counted and compared as exact, case-sensitive sets, so an answer repeated within a cell counts once; for the false-answer lists, a case-insensitive comparison gives the same counts. A row cites a web source when its Source cell begins with http, ignoring case. The best-incorrect column is compared with the listed false answers exactly, then ignoring case (no extra matches). The judge file is data/finetune_truth.jsonl at 8b48f68, the last commit that touched it; its label is the trimmed completion, and distinct questions are counted by the prompt text before the answer line. The median of an even number of counts averages the two middle values (both are 4 here). Percentages are rounded once, half up. The script (xs0219_truthfulqa_negatives.py) and its output, counts only, are kept in the writer's working folder.
Cost model: xs0219_validation_cost.py computes candidates as 1,000 divided by the confirmation rate, records examined as candidates divided by the base rate, screening hours as records times screening minutes over 60, validation hours as candidates times minutes times reviewers over 60, and lift per 100 hours from an invented lift per 1,000 accepted. Exact fractions, rounded once, half up; no randomness. All its inputs are examples.
Published evidence: the arXiv papers by Zhu and colleagues, Yuksekgonul and colleagues, Hsieh and colleagues and Lin and colleagues, verified through search excerpts on arxiv.org, which a reviewer should open before release, and the TruthfulQA repository and README, read directly. Not measured: lift from any business negative set; validation hours at any company, SourceX included; prices; and how fast business failure labels decay in practice.
Frequently asked questions
What is a negative example in AI training data?
It is a recorded case of what is wrong: a failed attempt, a near-miss that does not match, a plausible false answer or a real operational failure. It is useful when it records why it is wrong and who confirmed that. Without that documentation, a buyer cannot separate a true failure from a mislabeled success.
Are negative examples more valuable than positive ones?
Not as a rule. Published results show negatives can carry strong learning signal, for example in reinforcement learning with verifiable answers, but their value depends on measured lift for a specific failure and on what they cost to validate. A small, well-documented set can beat a large, cheap one, and the reverse is also possible.
Why do rare failure cases cost so much to collect?
Because most of the cost is finding them. If one record in 40 is a candidate and specialists reject 40 percent of candidates, 1,000 accepted cases mean screening about 66,667 records in the illustrative model above. Better source filters cut that cost more than faster reviewers do.
Can synthetic data replace documented negative examples?
Where an automatic checker can verify answers, generated failures are cheap and effective. Where judgment is needed, template-made negatives can mislead: SugarCrepe found earlier gains overestimated once tests used fluent, model-written negatives. Test any synthetic set against real, held-out failures.
How should an investor value a company selling failure data?
Ask for lift by family of negatives on held-out tests, validation hours per accepted case, rejection rates with reasons, a label version history and the rights each set carries. A company that prices by volume alone is not measuring what drives value.
Sources
- Zhu and colleagues report that reinforcement learning with only negative samples can be highly effective across Pass@k, while reinforcing only correct responses improves Pass@1 but degrades performance at higher k. Source
- Yuksekgonul and colleagues propose NegCLIP, composition-aware hard negative mining with perturbed-order captions and nearest-neighbor images, reporting gains on order-sensitive tasks. Source
- SugarCrepe rebuilds compositionality tests with model-written hard negatives and finds improvements from methods such as NegCLIP overestimated through overfitting to earlier benchmark artifacts. Source
- The TruthfulQA paper describes 817 questions in 38 categories; under human evaluation the best model was truthful on 58 percent of questions against 94 percent for humans. Source
- The TruthfulQA repository's release history of TruthfulQA.csv and its judge-training file, counted for this article at pinned commits. Source
- The TruthfulQA README says fine-tuned GPT-3 judges reach generally about 90-95 percent validation accuracy in predicting human evaluations of truthfulness and informativeness. Source
- SourceX's how-it-works page says licenses are exclusive for AI training and that a supplier confirms the data hasn't been licensed for AI training before. Source
- SourceX's investor page offers investor materials on request and keeps commercial metrics for investors. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.