AI data market
Durable AI training data demand: what outlives a model upgrade
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Durable AI training data demand comes from needs a stronger model recreates instead of removing: re-testing at every model change, records newer than the knowledge cutoff, test sets that must stay unseen, and private operating context. Demand that only patches today's weakness tends to end at the next upgrade. Meta's Llama 3 card cites over 10 million human-annotated tuning examples; the Llama 3.1 card, three months later, cites over 25 million synthetic ones.
Value drivers
Durable AI training data
Key takeaways
- A demand survives an upgrade when the upgrade itself triggers it: a forced re-test, a newer cutoff, a leaked test or new private records.
- Generic demonstrations and patches for a named weakness are the most exposed; developers increasingly fill both with synthetic data and in-house fixes.
- Model retirements create re-testing work, but a buyer only re-buys when its test set saturates, leaks or stops matching the business.
- Knowledge cutoffs and benchmark contamination are two refresh clocks no model upgrade resets.
- Durability is a survival rate by upgrade cohort, with open cohorts censored; no seller has published one.
Which AI training data demand lasts beyond the next model?
Demand lasts when the next model recreates the need: the model must be re-tested, it knows nothing after its cutoff, it can absorb public tests into training, and it still cannot see a company's private operations. Demand that exists only because the current model is weak at something tends to end when a stronger model, or its synthetic output, closes the gap.
For an investor, durable AI training data demand is the part of a data seller's revenue that a model upgrade renews rather than removes. The question is not whether models need data in general. It is whether a specific purchase is made again after the next release, by whom and on what evidence.
This article separates the two with an upgrade survival test, classes six common demands, explains the two refresh clocks that keep some of them recurring, and sets out how to measure survival by cohort, which no one has yet published.
Temporary weakness or recurring requirement: the upgrade survival test
Put four questions to any data purchase. A demand that answers yes to the first and no to the other three is temporary, however large it looks today.
The test turns on cause. A weakness belongs to one model and disappears when a better one arrives. A requirement belongs to the buyer's world: a release schedule, a calendar, a public test that leaks, a business that keeps producing private records. Requirements survive upgrades because the upgrade is what sets them off.
The four questions:
Checklist · 4 items
- Cause: does the need exist because the current model fails at something? If yes, assume a stronger model or its synthetic output can close it.
- Trigger: does a new model, a new quarter or a business event create the need again? Recurring triggers are the core of durable demand.
- Self-supply: can the model developer or the buyer produce the data with the new model? If yes, the purchase is exposed at every upgrade.
- Privacy of source: does the data come from records that are not publicly accessible? Private sources resist substitution for as long as the business behind them keeps generating them.
0 of 4 ticked. Tick each item as you go.
Six data demands ranked by upgrade survival
Two demands look temporary, three recur on clocks set outside the model, and one lasts as long as its source stays private. The figure lays the same six across two upgrades.
Classes are editorial, assigned on 2026-10-26 from the cited sources; they describe how an upgrade acts on each demand, not how large each one is. The last column is the private evidence a seller should be able to show an investor.
The six demands and the evidence behind each class:
| Demand | Survival class | Dated evidence | What renews it | Ask a seller for |
|---|---|---|---|---|
| Generic instruction demonstrations | Temporary: replaced by synthetic examples | Llama 3 card, April 2024: over 10M human-annotated tuning examples; Llama 3.1 card, July 2024: over 25M synthetic ones | Nothing once a model writes good examples | Revenue share from generic demonstrations, by quarter |
| Patches for a named model weakness | Temporary: developers plan in-house fixes | DeepSeek-R1 paper: software engineering gap named, with future versions to address it through rejection sampling or asynchronous evaluation | A new weakness, not the old one | Orders that ended after the weakness closed |
| Regression and acceptance evaluations | Recurring: every model change forces a re-test | Anthropic deprecations page: at least 60 days notice before retirement; Claude Opus 4.1 retired August 5, 2026 | Model retirements and upgrades | Replacement-task orders tied to named releases |
| Records newer than the knowledge cutoff | Recurring: each model is a static snapshot | Llama 4 card, April 2025: pretraining data cutoff of August 2024 | The calendar and business change | Refresh deliveries with dates and record counts |
| Unseen holdout test sets | Recurring: public tests leak and saturate | LiveBench: questions updated monthly, new ones withheld; OpenAI stopped reporting SWE-bench Verified | Contamination and saturation | Retired test items and replacement cadence |
| Private operating context | Durable while the source stays private | OpenAI data partnerships page: seeks datasets not already easily accessible online | New records from ongoing operations | Share of buyers renewing after a model upgrade |
Why model retirements create re-testing, not automatic re-buying
Every forced model change creates evaluation work, but only a worn-out test set creates a new purchase. An investor who reads retirement schedules as recurring revenue should check which of the two a seller actually captures.
Retirement schedules are now explicit. Anthropic's model deprecations page says it gives at least 60 days' notice before retiring a publicly released model, that requests to retired models fail, and that customers should test their applications with replacement models well before the retirement date. Its table shows claude-opus-4-1-20250805 retired on August 5, 2026, a year to the day after the date in its model name, and claude-sonnet-4-20250514 retired on June 15, 2026. OpenAI's deprecations page lists a July 14, 2025 shutdown for gpt-4.5-preview, with gpt-4.1 as the recommended replacement.
Evaluation practice assumes the re-test happens every time. Anthropic's January 2026 guidance on agent evaluations recommends running automated evals on each agent change and model upgrade, and says capability evals with high pass rates can graduate into a regression suite. It defines saturation as the point where an agent passes every solvable task, leaving no room to measure improvement.
That distinction sets the revenue model. A regression suite that still separates good answers from bad is rerun, not rebought; the buyer pays again when the suite saturates, when its tasks leak or when the business changes what counts as correct. Durable evaluation revenue therefore shows up as replacement tasks and refresh terms, not as repeat sales of the same set.
Two refresh clocks: knowledge cutoffs and leaking tests
Two clocks keep some demand recurring however capable the next model is: the date its training data stops, and the speed at which public test items reach training data.
The first clock is printed on model cards. Meta describes each Llama release as a static model trained on an offline dataset. The Llama 3 70B model, released April 18, 2024, has a pretraining cutoff of December 2023; Llama 4, released April 5, 2025, has a cutoff of August 2024. Each upgrade moves the cutoff forward, but never to the present, so recent events, prices, policies and procedures stay outside the weights.
Licensing contracts already price this clock. Reddit's annual report for 2024 says the data licensing arrangements it entered in January 2024 deliver content through continuous access to its data API and quarterly transfers of data over the term, and that remaining performance obligations of $252.9 million at December 31, 2024 consisted primarily of long-term content licensing contracts. The buyers pay for a flow of new records, not only a stock.
The second clock runs on contamination. LiveBench was built with questions updated monthly from recent sources and scored against objective answers, and its authors now withhold each month's new questions as the benchmark gains users. OpenAI stopped reporting SWE-bench Verified after an audit of a subset that models often failed found that at least 59.4 percent of the problems reviewed had tests that reject correct solutions, and after concluding that gains no longer reflected real software development ability. A test that leaks or breaks has to be replaced, and replacement is a purchase.
Could stronger models end outside data spending altogether?
For temporary demands, largely yes; for recurring and private ones, the public evidence points the other way. The strongest alternative explanation, that capability progress and model announcements never reach a data seller's revenue, deserves a fair hearing.
Synthetic substitution is already visible inside one model family. Meta's Llama 3 card cites over 10 million human-annotated fine-tuning examples; the Llama 3.1 card, published three months later, cites over 25 million synthetically generated ones, while noting that Meta still combines human-generated data from vendors with synthetic data. The same card says its license allows using the model's outputs to improve other models, including through synthetic data generation and distillation.
Developers also close named gaps themselves. The DeepSeek-R1 paper names software engineering as a weakness and says future versions will address it through rejection sampling on software engineering data or asynchronous evaluations during reinforcement learning, both in-house methods.
What synthetic data does not reset is the clocks. A model cannot generate records of events after its cutoff, cannot vouch that a test it may have trained on is unseen, and cannot produce a company's private operating history. If a seller's revenue sits in those categories, upgrades renew it; if it sits in generic demonstrations and weakness patches, upgrades wear it down.
How to measure upgrade survival by cohort
Group purchases by the model release they were bought for, then check whether each was re-requested, re-evaluated and re-committed after the next release. That turns durability from a story into a survival rate.
At a glance
How to measure upgrade survival by cohort
The design: for each purchase, ask whether the same buyer requested, evaluated and committed to a refresh or replacement after the next model change. Interview independent budget owners at model developers and at companies that fine-tune or evaluate models about their own purchase records, excluding anyone with a commercial relationship to the interviewer. Credit each renewal to the upgrade that triggered it.
Use purchases in the cohort as the denominator. Purchases whose next upgrade has not happened yet are censored, counted neither as survivors nor as losses, and below 30 purchases results are reported as counts and ranges rather than percentages. SourceX has not run or published this study, and no survival rate for any demand class is reported here. A result would have to place each purchase on these levels:
Checklist · 5 items
- Level 1, bought once: a purchase tied to one model version, with a date and an acceptance test.
- Level 2, re-requested: a written request for a refresh or replacement after the next model release, from the same budget owner.
- Level 3, re-evaluated: the refreshed data tested on the new model, with the result compared to the original.
- Level 4, re-committed: a signed order for the refresh, or a contract with refresh terms.
- Level 5, survived twice: re-committed after two consecutive model releases. Only this level separates a durable demand from a lucky renewal.
0 of 5 ticked. Tick each item as you go.
Risks that could make the durability thesis wrong
Five developments could shorten the life of the demands classed here as recurring or durable. Each comes with a signal worth tracking.
- Vendors could stop forcing upgrades. Anthropic's deprecations page says it has committed to long-term preservation of model weights and hopes to make past models publicly available again; longer model lives would mean fewer forced re-tests.
- Retrieval could absorb freshness. If buyers pass recent records to models at run time, refresh spending moves from training licenses to access arrangements with different pricing and rights terms.
- Neutral test hosts could make public benchmarks last. LiveBench already withholds new questions; if hosted private tests stay unseen at scale, fewer buyers need replacements of their own.
- Concentration. Reddit's annual report for 2024 says "substantially all of the contract value associated with our licensing revenue is derived from two of our partners"; a seller whose refreshes depend on one or two buyers faces renewal risk at every upgrade.
- Private context may never leave the building. Companies that deploy models on their own records may not license them to anyone, which caps the third-party market even where the demand is durable.
Questions to ask a data seller about upgrade survival, SourceX included
Ask which class each revenue line falls in and what happened to it at the last model release, and expect SourceX to answer on the same terms.
For SourceX, the question lands on licensed business records. Working for companies that license operational data to AI developers, it takes on qualification and sourcing, the rights review, contracts, preparation of the data, delivery and payment. It has published no renewal, refresh or survival figures.
Only verified accredited investors can take part in its current Rule 506(c) raise. Its investor materials, including the company and commercial metrics it keeps out of public articles, are available by request through the form at sourcex.si/investors.
Checklist · 6 items
- Which of the six demand classes produced each revenue line over the last four quarters?
- Which orders ended after a named model release, and why?
- For refresh contracts, what are the cadence, the record counts per delivery and the renewal terms?
- What share of buyers re-requested data after their last model change, with open cohorts reported as censored?
- How do you show that a holdout set you sell has not leaked into training data?
- What share of revenue comes from your two largest buyers?
0 of 6 ticked. Tick each item as you go.
Methods note: how the survival classes were assigned
The upgrade survival test, the six survival classes, the survival ladder and the cohort design were put together on 2026-10-26 as this article's editorial frameworks; the evidence is drawn from Meta's Llama 3, 3.1 and 4 model cards, the DeepSeek-R1 paper, Anthropic's model deprecations page and January 2026 evaluation guidance, OpenAI's deprecations page, data partnerships page and SWE-bench Verified notice, the LiveBench paper and Reddit's annual report for 2024.
Not measured: renewal or survival rates for any demand class, spending by class at any model developer, SourceX's own orders or renewals, and the effect of retrieval on refresh demand. The classes describe how an upgrade acts on a demand; they do not size it, and the timeline in the figure is illustrative.
Frequently asked questions
What is durable AI training data demand?
It is demand that a model upgrade renews rather than removes: re-testing at each model change, records newer than the model's knowledge cutoff, replacement test sets when public ones leak, and private operating context. Demand that exists only to fix a current model's weakness usually ends when a stronger model or its synthetic output closes the gap.
Do model upgrades reduce the need for human-labeled data?
For generic instruction data, the public evidence says yes. Meta's Llama 3 card cites over 10 million human-annotated fine-tuning examples, while the Llama 3.1 card three months later cites over 25 million synthetic ones, though Meta still combines vendor-supplied human data with synthetic data.
Why do companies need new evaluation data when they switch models?
Because the new model can fail where the old one passed, and providers retire models on a schedule. Anthropic gives at least 60 days' notice before retiring a public model and advises testing replacements well before that date. An existing suite can be rerun; it has to be replaced when it saturates, leaks or no longer matches the business.
How often does training data need refreshing?
No single cadence fits, because refresh follows triggers rather than a calendar alone. Each model is a static snapshot with a cutoff, such as August 2024 for Llama 4, so records of later events need fresh delivery; Reddit's licensing arrangements, for example, include quarterly data transfers over their term.
Does SourceX publish renewal or refresh data?
No. SourceX has published no renewal, refresh or survival figures and shares company and commercial metrics with prospective investors directly. Materials are requested through sourcex.si/investors, and its Rule 506(c) raise admits verified accredited investors alone.
Sources
- Meta's Llama 3 model card (released April 18, 2024) describes a static model trained on an offline dataset, fine-tuning data with over 10M human-annotated examples, and a December 2023 cutoff for the 70B model. Source
- Meta's Llama 3.1 model card (July 23, 2024) cites over 25M synthetically generated fine-tuning examples, says human vendor data is combined with synthetic data, and says the license allows outputs to be used for synthetic data generation and distillation. Source
- Meta's Llama 4 model card (April 5, 2025) gives a pretraining data cutoff of August 2024 for a static model trained on an offline dataset. Source
- The DeepSeek-R1 paper names software engineering as a weakness and says future versions will address it through rejection sampling or asynchronous evaluations. Source
- Anthropic gives at least 60 days' notice before retiring publicly released models, advises testing replacements well before retirement, lists Claude Opus 4.1 as retired August 5, 2026, and has committed to long-term preservation of model weights. Source
- OpenAI's deprecations page lists a July 14, 2025 shutdown for gpt-4.5-preview with gpt-4.1 as the recommended replacement. Source
- Anthropic's January 2026 guidance recommends running evals on each model upgrade, says capability evals with high pass rates can graduate to regression suites, and defines eval saturation. Source
- The LiveBench paper describes questions updated monthly from recent sources, scored against objective ground truth, with new questions withheld as the benchmark grows. Source
- OpenAI stopped reporting SWE-bench Verified after finding that at least 59.4% of audited problems had tests rejecting correct solutions and that gains no longer reflected real-world ability. Source
- OpenAI's data partnerships page seeks large-scale datasets that are not already easily accessible online. Source
- Reddit's annual report for 2024 describes licensing delivered through continuous API access and quarterly data transfers, remaining performance obligations of $252.9 million mostly from long-term content licensing, and substantially all licensing contract value derived from two partners. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.