AI data market
Evaluation versus training data: a flywheel test of the market's future
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Evaluation could outgrow training data licensing only if its purchase triggers keep recurring and buyers pay outsiders to meet them: model releases, contamination, regulation, new deployments, reliability demands and independent assurance. No public data measures either market, so the evaluation versus training market future is a conditional bet. Track paid repurchases tied to dated triggers, and watch whether free tooling, synthetic tests or buyer concentration break the loop.
Evaluation split
Evaluation
Key takeaways
- Evaluation can outgrow training data licensing only through repeat purchases, because test sets lose value with exposure while training data is absorbed once.
- Keep training licenses, evaluation data, expert labor and tooling as separate budgets; adding them together inflates every estimate.
- The flywheel test asks two things of each of six triggers: does it recur, and does the buyer pay an outside supplier to meet it?
- Customer-specific test programs built from a company's own records have the strongest recurrence logic; private static benchmarks decay fastest.
- Free tooling such as Inspect, synthetic tests, concentrated buyers and curator conflicts of interest are the risks that could stop the flywheel.
Could evaluation spending outgrow training data licensing?
It could, but only if evaluation keeps being bought again for reasons training data does not share. A training dataset is largely used up once its signal is absorbed into a model. An evaluation set loses value as it is exposed, and every new release, deployment or rule creates a fresh reason to buy. If those reasons recur faster than training data is refreshed, evaluation spending could pass training data licensing even though neither market is measured today.
That is a conditional thesis, and this article treats it as one. Training data licensing occasionally shows up in public filings; evaluation work mostly hides inside research budgets, vendor services and staff time. So no market sizes or shares appear here. Instead the article separates the budgets, builds a test of the evaluation flywheel from dated public signals, sets out four scenarios with signposts, and names the evidence that would prove the idea wrong.
The question matters because the two markets reward different businesses. Training data licensing rewards rights and scale. Evaluation rewards freshness, independence and the ability to keep test material unseen, which favors suppliers of records that have never been on the public web.
Three budgets investors should not add together
Training data licensing, evaluation data and the expert labor behind both are separate budgets with separate purchase triggers, and adding them together is how most market estimates inflate. The table defines each one by what is bought and how its value is used up.
Keep annotation and tooling outside both definitions. Expert hours can produce a training set or a test set, and counting them twice inflates both. Tooling is software, not data: it lowers the cost of running evaluations but supplies none of the unseen tasks a release decision needs. The licensing difference itself, training rights against evaluation-only rights, is covered on this site's comparison of the two license types.
| Budget | What is bought | How its value is used up | Typical purchase trigger | Where it shows up publicly |
|---|---|---|---|---|
| Training data licensing | Rights to use records or content to change model weights | Absorbed into the model; repeated use adds little | A new model generation or a capability gap | Occasionally in filings, such as Reddit's 2024 S-1 |
| Evaluation data | Held-out tasks, reference answers and graders that measure a model or agent | Decays as items leak into training data or stop separating models | A release decision, deployment, audit or rule | Rarely; usually inside research or vendor spending |
| Expert annotation and review | Human time to write, label or judge items | Spent per task; can serve either budget | Any project that needs human judgment | Bundled into services revenue |
| Evaluation tooling | Software to run, score and log evaluations | Reusable; little marginal cost once built | Team size and evaluation volume | Open-source projects and platform pricing |
The Evaluation Demand Flywheel Test: six triggers and their breaking points
The flywheel turns only if each purchase trigger both recurs and is paid for outside the buyer's own team. The test lists six triggers, a dated public signal for each, and the evidence that would show the trigger is not producing repeat purchases. A trigger with no paid repurchase counts against the thesis even if the activity behind it is growing.
The triggers also feed each other, which is what makes this a flywheel rather than a list. Releases expose test sets and speed up contamination; contamination forces new sets; new deployments bring rules and audits; reliability demands multiply trials on every set. The loop breaks wherever buyers can meet a trigger with free, synthetic or self-built tests.
Two signals deserve weight because they come from the people who build models. In a post explaining why it no longer evaluates SWE-bench Verified (read October 26, 2026), OpenAI says the benchmark is increasingly contaminated and mismeasures frontier coding progress, and that an audit found at least 59.4 percent of audited problems had flawed test cases that reject functionally correct submissions. And OpenAI's GDPval (2025) measures models on graded tasks from 44 occupations, evidence that model builders now invest in evaluation shaped like real work.
Score the test by counting triggers, not by weighing anecdotes. For each row, mark paid repurchase observed, activity without paid repurchase, or no evidence, and date the mark. Fewer than three rows with observed paid repurchase means the flywheel is not turning as a market, however much evaluation activity there is; five or six means the thesis is passing its own test. The rule is deliberately coarse, because a finer score would imply a precision the public record cannot support.
The figure draws the loop and the three forces that could break it. The table is the checklist an investor can rerun each quarter: for every row, ask whether paid repurchase was observed, and keep the answer with its date.
| Purchase trigger | Recurs when | Dated public signal | Disconfirming evidence |
|---|---|---|---|
| Model releases | Each release needs a go or no-go decision on tasks the model has not seen | OpenAI's GDPval (2025) measures models on graded tasks from 44 occupations | Releases judged on public benchmarks alone, with no private test purchases |
| Contamination and saturation | Exposed or solved test sets stop separating models | OpenAI, read October 2026: SWE-bench Verified increasingly contaminated; 59.4 percent of audited problems had flawed tests | Static sets keep separating models for years without refresh |
| Regulation and audit | Rules require documented evaluation before or after release | EU guidelines for systemic-risk general-purpose models: evaluation with state-of-the-art tools, including adversarial testing | Obligations stay with a few frontier providers who build tests in-house |
| New enterprise deployments | Each workflow needs tests drawn from its own records | GDPval notes it is limited to one-shot tasks, without context building or multiple drafts | Enterprises accept vendor scores instead of testing on their own tasks |
| Reliability requirements | Agents must succeed repeatedly, multiplying trials per item | tau-bench repository: one setup fell from 0.460 at pass^1 to 0.225 at pass^4 | Reliability checked in production monitoring, not with bought test sets |
| Independent assurance | Buyers want tests the model builder did not write | Scale says its SEAL leaderboards use private datasets that cannot be gamed | Conflict-of-interest concerns push labs back to self-evaluation |
Static benchmarks, dynamic environments or customer test programs: which one recurs?
Customer-specific test programs have the strongest recurrence logic, because every new workflow, policy change or model swap creates a new test; static benchmarks have the weakest, because they decay with use, and public ones are never bought at all. Dynamic benchmarks sit between them: they refresh by design but need a steady supply of new source material.
Contamination is real but uneven, which matters for how fast static sets decay. A 2024 study comparing models on GSM8k, a popular grade-school math benchmark, with GSM1k, a new counterpart, found accuracy drops of up to 8 percent in its current version, with several model families showing systematic overfitting, while the frontier models it tested showed minimal signs of overfitting. Decay is fastest for the models least able to pay for private tests.
Illustrative example: an insurer deploying a claims-handling agent builds 400 test tasks from closed claims, each with the adjuster's final decision as the reference answer. Six months later a change in policy wording makes a quarter of those tasks wrong under the new rules, and a new product line adds a workflow the set never covered. Both events create a purchase no public benchmark can satisfy, which is why this product type has the strongest recurrence logic and also the heaviest rights burden: every task comes from records someone must be allowed to use for testing.
Trust cuts both ways. A 2025 arXiv paper on private evaluation curators argues that keeping prompts unseen reduces contamination, but that a curator who also supplies training data to the labs it evaluates faces a conflict of interest, and that expert annotators' preferences can tilt results toward models trained on the curator's own data. An evaluation supplier's independence is therefore part of its product, not a footnote.
| Evaluation product | Example | Why buyers come back | How it decays | Trust obstacle |
|---|---|---|---|---|
| Public static benchmark | SWE-bench Verified, which OpenAI stopped evaluating after finding contamination | No purchase: it is free, which is why it anchors comparisons | Fastest, because items are public from day one | Training on the items inflates scores |
| Private static benchmark | Scale's SEAL leaderboards, which the company says use private datasets | Each new model needs ranking on unseen items | Items leak through repeated use and saturate | The curator may also sell training data to the labs it ranks |
| Dynamic benchmark or environment | LiveBench, which releases new questions monthly to limit contamination | Fresh items arrive each cycle by design | Slowly, but every refresh needs new source material | Scores are harder to compare across releases |
| Customer-specific test program | Tasks built from one company's own records for one deployment | Every new workflow, policy change or model swap | Workflows and policies change, so tasks go stale | Needs rights to the records and a neutral grader |
How to judge the evaluation versus training market future without a market size
Judge the evaluation versus training market future by counting purchase events and their triggers, not dollars. Public dollar figures exist for a handful of training licenses and almost none for evaluation, so any size comparison today would compare a disclosed number with a guess.
At a glance
How to judge the evaluation versus training market future without a market size
The training side shows how thin even the best disclosures are. Reddit's S-1 (2024) reported $203.0 million of aggregate contract value for data licensing arrangements signed in January 2024, with terms of two to three years. Its annual report for 2024 then says that, so far, two partners are the source of substantially all of the contract value linked to its licensing revenue. One company, two buyers: precise facts, but not a market.
Evaluation leaves even fewer traces, so the evidence has to be built from events. The measures below can be collected from contracts, announcements and interviews, compared over time, and falsified. SourceX has not published these measures for its own business; it shares company and commercial metrics with investors directly.
Checklist · 6 items
- Repurchase interval: days between successive evaluation purchases by the same buyer, with open intervals counted rather than dropped.
- Trigger attribution: the share of evaluation purchases tied to a dated release, contamination finding, rule or deployment.
- Distinct buyers: how many separate organizations bought held-out test material in a period, and the share from the largest two.
- Self-build share: the share of a buyer's evaluations written in-house, from public engineering posts and interviews.
- Refresh ratio: refreshed items delivered as a share of the original set, by product type.
- Training renewal rate: training licenses renewed at term end, the comparison series for the same period.
0 of 6 ticked. Tick each item as you go.
Four scenarios, each with a signpost and a kill condition
Four scenarios cover the plausible paths, and none is assigned a probability. Each names what must hold, a signpost an investor can watch, and the evidence that would end it. Update by moving weight between scenarios when a signpost fires, not by inventing numbers.
Epoch AI's forecasts anchor the last row. Its 2024 study estimated that roughly 300 trillion tokens of quality, repetition-adjusted public human text make up the effective stock, and projected full use between 2026 and 2032, earlier with heavy overtraining. Its later outlook, What will AI look like in 2030?, finds enough public text to scale to at least 2027, notes that synthetic data can be generated in large quantities and calls a data bottleneck hard to rule out but surmountable.
| Scenario | What must hold | Signpost to watch | What would end it |
|---|---|---|---|
| Evaluation-led | At least four of the six triggers show paid repurchase, and unseen test material stays scarce | Model builders retire public benchmarks faster than new ones gain trust | Open, refreshed benchmarks prove good enough for release decisions |
| Parallel growth | Evaluation repurchases rise while each training generation still buys new data | Filings and announcements keep naming training licenses alongside evaluation programs | Concentrated training contracts lapse without renewal |
| Commoditized evaluation | Open tooling, synthetic tests and model graders cover most needs | Free evaluation suites expand and enterprises adopt them unchanged | Synthetic tests fail to predict production failures, forcing tests built from real records |
| Both shrink | Models need far less outside data of either kind | Forecasts move further toward surmountable data limits | Capability gaps on private business work persist |
What could make the evaluation thesis wrong
The thesis fails if buyers can meet evaluation triggers cheaply on their own, or if too few buyers exist to sustain repeat purchases. Each risk below has a public marker an investor can check.
At a glance
What could make the evaluation thesis wrong
None of these risks is hypothetical, and several already have strong evidence behind them. The honest reading is that evaluation demand is likely to keep growing as activity, while the share paid to outside data suppliers is the open question.
- Free tooling: Inspect, the evaluation framework from the UK AI Security Institute, includes over 200 pre-built evaluations ready to run on any model (README, read October 26, 2026). Tooling lowers the cost of running tests, though not of writing unseen ones.
- Synthetic and model-written tests: Epoch AI judges a data bottleneck surmountable, partly through synthetic data. If generated tasks predict production failures well enough, paid test material loses its premium.
- Buyer concentration: two partners account for substantially all of Reddit's licensing contract value. Evaluation buyers may be just as concentrated among a few labs, making revenue lumpy.
- Uneven contamination: GSM1k found minimal overfitting in the frontier models it tested, so the contamination trigger may be weakest for the buyers with the largest budgets.
- Narrow regulation: the EU's systemic-risk evaluation duties apply to the most capable general-purpose models, with a presumption above 10^25 training FLOP in the Commission's guidelines, not to every deployment.
- Conflicts of interest: if curators that sell training data also run evaluations, buyers may discount their results or build tests in-house, as the 2025 private-curator paper warns.
Questions to ask any company selling evaluation data, SourceX included
Ask for evidence of repurchase, triggers and independence, defined so the answers can be checked. The same questions apply to any company that licenses data for testing models, SourceX included.
On the supply side, SourceX handles each step for businesses licensing operational data to AI developers: it sources and qualifies the data, reviews the rights, does the contracting, prepares the records, delivers them and settles payment. It has not published how its business divides between training and evaluation uses; its company and commercial metrics go to investors directly rather than into articles. The company's current raise, under Rule 506(c), is for accredited investors, and investor materials are available on request through the investors page.
Checklist · 6 items
- By contract count, how many licenses permit evaluation only, training only, or both, and how is each use defined in the agreement?
- What dated event triggered each repeat evaluation purchase: a release, a contamination finding, a rule or a new deployment?
- How is held-out status protected after delivery, and what happens to a test set when a buyer reports leakage?
- How many distinct buyers made repeat evaluation purchases in the period, and what share came from the largest two?
- Who writes and grades the tasks, and does the seller also supply training data to the same buyer?
- Which customer-specific programs were renewed after a workflow or policy change, and which lapsed?
0 of 6 ticked. Tick each item as you go.
Methods note: how the flywheel triggers were chosen and checked
The Evaluation Demand Flywheel Test and the four scenarios are editorial frameworks built for this article. Triggers were chosen because each has a dated public signal, and the disconfirming evidence for each was written before any trigger was judged.
Sources were checked on October 26, 2026: OpenAI's GDPval and SWE-bench Verified posts, the European Commission's guidelines for general-purpose AI providers, two arXiv papers (GSM1k and the private-curator paper), the tau-bench, LiveBench and Inspect repositories, Scale's leaderboard announcement (a company claim, attributed as such), Epoch AI's data forecasts and Reddit's SEC filings.
Not measured: dollar spending on evaluation or training data, market shares, prices, growth rates, scenario probabilities, or any SourceX data. Repository statistics change; tau-bench's maintainers now point users to a newer version of the benchmark, and Inspect's evaluation count will move.
Frequently asked questions
Is evaluation data a bigger market than training data today?
No public data shows that, and neither market is measured cleanly. Training data licensing occasionally appears in filings, such as Reddit's 2024 S-1, while evaluation spending is usually bundled into research budgets and services. Any claim that one is larger should state its definitions and sources.
Why does evaluation data have to be refreshed?
Test items leak into training data or get solved, and then they stop separating models. OpenAI says SWE-bench Verified became increasingly contaminated, and LiveBench releases new questions monthly to limit contamination. Refresh is what turns a one-time test set into a repeat purchase.
Can the same dataset be used for training and evaluation?
Not for the same model at the same time: a test set that enters training stops measuring anything. That is why evaluation sets are kept apart from training data, and a license can permit evaluation use without permitting training.
What evidence would show the evaluation flywheel is real?
Paid repurchases tied to dated triggers across several distinct buyers, customer test programs renewed after workflow or policy changes, and private test material that keeps its value after free evaluation tooling becomes widespread. Activity growth alone does not count; outside purchases do.
Does SourceX publish how much of its business is evaluation data?
No. SourceX has not published how its business divides between training and evaluation uses and shares company and commercial metrics with investors directly. Investors can ask for materials through sourcex.si/investors and put the training-versus-evaluation question to the company; any offering is open solely to verified accredited investors.
Sources
- OpenAI says SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress, and its audit found at least 59.4% of audited problems had flawed test cases that reject functionally correct submissions. Source
- OpenAI's GDPval measures models on tasks from 44 occupations in the top nine industries contributing to U.S. GDP and is limited to one-shot evaluations that do not capture context building or multiple drafts. Source
- The European Commission's guidelines for general-purpose AI providers list model evaluation with standardised protocols and state-of-the-art tools, including adversarial testing, among the obligations for models with systemic risk, presumed above 10^25 FLOP of training compute. Source
- The GSM1k study (arXiv 2405.00332, current version) found accuracy drops of up to 8% for some models, with several families showing systematic overfitting, and minimal signs of overfitting in frontier models. Source
- Reddit's S-1 disclosed data licensing arrangements entered in January 2024 with an aggregate contract value of $203.0 million and terms of two to three years. Source
- Reddit's annual report for 2024 says substantially all contract value associated with its licensing revenue comes from two partners. Source
- Scale says its SEAL leaderboards rank frontier models using curated private datasets that cannot be gamed. Source
- A 2025 arXiv paper on private evaluation curators argues that private evaluations reduce contamination but raise conflict-of-interest risks when curators have business relationships with the labs they evaluate, and that annotator preferences can bias results. Source
- LiveBench is designed to limit potential contamination by releasing new questions monthly. Source
- Inspect, the evaluation framework created by the UK AI Security Institute, includes over 200 pre-built evaluations ready to run on any model. Source
- The original tau-bench repository's airline leaderboard lists one tool-calling setup at 0.460 pass^1 and 0.225 pass^4. Source
- Epoch AI estimated the effective stock of quality, repetition-adjusted public human text at about 300 trillion tokens and projected full use between 2026 and 2032, earlier with heavy overtraining. Source
- Epoch AI's 2030 outlook finds enough public human text to scale to at least 2027, notes synthetic data can be generated in large quantities, and calls a data bottleneck difficult to rule out but surmountable. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.