AI data market
Evaluation data investment opportunity: 6 tests of release-gate demand
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Evaluation data can matter before training data because a release decision needs a held-out test before anyone knows which training data is worth buying. Public evidence shows the gates are real: pre-release government testing agreements since 2024, an EU duty to evaluate the most capable models, and lab policies revised repeatedly. Whether gates pay outside suppliers is unproven, so treat the evaluation data investment opportunity as a testable thesis.
Evaluation split
Evaluation data investment
Key takeaways
- A release decision needs a held-out test before anyone knows which training data to buy, so evaluation enters the cycle first and returns at every gate.
- Six tests separate a release-gate demand pool from training demand: trigger, timing, unit of purchase, rights scope, refresh clock and budget owner.
- Public evidence shows the gates are real: pre-release government testing since 2024, an EU evaluation duty for the most capable models, and lab policies that keep changing.
- Training data shows public prices without public refresh; evaluation shows public refresh without public prices. That gap is the diligence question.
- In-house suites, free public benchmarks and a small set of regulated buyers could keep the pool real but unpaid.
Why can evaluation data matter before training data?
Because a release decision needs a test before anyone knows which training data is worth buying. The test defines what failure looks like, measures how far the current model is from passing and decides whether a new version ships. Training data is bought to close a gap the test has already measured, so within one release, evaluation comes first and comes back at every gate.
Anthropic's engineering guide to agent evaluations, published January 9, 2026, describes that order. It recommends building evaluations that define planned capabilities before an agent can perform them, calls 20 to 50 simple tasks drawn from real failures a great start, and observes that when a stronger model arrives, teams without evaluations face weeks of testing while teams with them can upgrade in days.
For an investor, the order matters only if it creates spending that training budgets do not already cover. This article tests that narrower claim: whether release decisions form a distinct demand pool with its own trigger, buyer, rights and refresh clock, how far public evidence climbs toward paid commitments, and what would show the pool is real but unpaid.
Where does evaluation data enter a model release cycle?
At almost every stage, while training data usually enters at one. Map a single release and the asymmetry is plain: the test set is written before training, used to decide what to train on, rerun on each checkpoint, used again at the release gate, kept as a regression suite after launch and rerun when the next model arrives.
Anthropic's guide supplies most of the stages. Automated evaluations run on each agent change and model upgrade before launch, and production monitoring takes over afterward. Capability evaluations with high pass rates graduate into a regression suite that runs continuously to catch drift, and when a new model drops, rerunning the suite shows which bets paid off. The table compresses that sequence into six stages; the training column is editorial and assumes one purchase per model generation.
| Stage | What evaluation data does | What training data does | What makes the evaluation stale |
|---|---|---|---|
| 1. Define the target | Turns observed failures into tasks with pass or fail rules | Not bought yet; the gap is unmeasured | Workflow or policy changes |
| 2. Baseline the current model | Measures the gap on held-out tasks | Purchase scoped against the measured gap | Items leaking into public data |
| 3. Train or tune | Scores each checkpoint for regressions | Consumed once in the training run | Repeated use that invites overfitting |
| 4. Release gate | Decides ship or hold, sometimes with outside testers | No role at the gate | New rules and revised lab policies |
| 5. After launch | Graduates into a regression suite run continuously | Production feedback may seed the next purchase | Scores saturating near the top |
| 6. Next model arrives | Reruns to compare the old and new models | A new generation may need new data | Capabilities outgrowing the tasks |
Six tests that separate a release-gate pool from training demand
Release decisions form a distinct demand pool only if evaluation data differs from training data on all six tests below: what triggers a purchase, when it happens, what one unit is, which rights it needs, what makes it stale and who signs for it. If evaluation shares its trigger, budget and rights with training, it is a line inside the training budget, not a separate market.
Public sources support a difference on most rows. Anthropic's guide treats a few dozen tasks from real failures as a sound starting set, while OpenAI's data partnerships page goes the other way, seeking large-scale datasets that the public cannot already reach easily online. A test set works only while models have not seen it, so its license can be narrower than a training grant; this site's guide to evaluation-only license terms covers that clause.
The weakest row is the budget owner. No public source found for this article shows who signs for evaluation data at a model developer, or whether that money is separate from the training data line, which is why the interview design later in the article starts there. Read the last column as the falsification test for each row.
| Test | Release-gate evaluation data | Training data | The pool is not distinct if |
|---|---|---|---|
| Trigger | A release, model swap, new rule or contamination finding | A capability gap or a new model generation | Evaluation purchases only ever follow training purchases |
| Timing | Before training, then again at every gate | Once per training run | Test sets are built after the data is bought |
| Unit of purchase | From 20 to 50 starter tasks (Anthropic) to 1,320 expert tasks (GDPval) | Large-scale records or content | Buyers ask for training-sized volumes to test with |
| Rights scope | Use for testing only, kept out of training | Use that changes model weights | Every evaluation license also grants training |
| Refresh clock | Set by releases, saturation and leakage | Set by model generations and contract terms | Test sets stay useful across several model generations |
| Budget owner | Evaluation, release or model risk teams | Data acquisition or research teams | One owner signs for both from one budget |
Do release decisions refresh evaluation data on their own clock?
Yes, and on a shorter horizon than public training licenses describe. Test sets go stale through use, leakage and capability growth, all of which follow the release cadence rather than the length of a data contract. The clearest evidence comes from model builders and evaluators retiring or rebuilding their own tests.
Saturation is the first clock. Anthropic's January 2026 guide notes that SWE-bench Verified scores had started at 30 percent within the year and that frontier models were nearing saturation above 80 percent. OpenAI's post explaining why it no longer evaluates SWE-bench Verified (read October 26, 2026) reports that progress slowed from 74.9 to 80.9 percent over six months, that at least 59.4 percent of the problems in its audited subset had flawed tests, and that models could reproduce solution details from a task identifier alone.
Capability growth is the second. METR reported in March 2025 that the length of tasks AI agents can complete with 50 percent reliability had doubled roughly every seven months for six years. Its Time Horizon 1.1 update in January 2026 expanded the task suite from 170 to 228 tasks and roughly doubled the number of tasks estimated at eight hours or longer. A test that measures the frontier has to grow with it.
Maintenance is the third. The tau-bench repository now warns that its airline and retail tasks are outdated and sends users to a successor with fixed tasks and a new banking domain. Release policies move as well: Anthropic's Responsible Scaling Policy page lists nine versions between September 19, 2023 and July 8, 2026. Training licenses run on a different horizon; Reddit's annual report for 2024 describes its content agreements as medium-term in length.
How far does public evidence take the evaluation data investment opportunity?
To the edge of payment and no further. On a five-rung ladder applied to release gates, public evidence shows a trigger on record, outside parties testing before release, a need for unseen material and test sets being rebuilt. No public filing or announcement found for this article discloses a paid commitment for evaluation data with a stated value and term.
At a glance
How far does public evidence take the evaluation data investment opportunity
Training data climbs the same ladder in a different shape. OpenAI's data partnerships page invites outside partners to supply datasets not already easily accessible online, which covers rungs 1 to 3. Reddit's annual report for 2024 reaches rung 4: it discloses content licensing with contract value and states that "to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners." No public source shows a renewal or refresh. Training data shows prices without refresh; evaluation shows refresh without prices.
Count a rung as reached only with a dated public source. Rung 4 is the one that would turn an operating pattern into an investable market, and it is empty for release gates. The figure sets the two profiles side by side; the ladder for release gates is below:
- Rung 1, a trigger on record: the EU AI Act requires providers of general-purpose AI models with systemic risk to evaluate them using state-of-the-art protocols and tools, including documented adversarial testing, and presumes systemic risk when training compute exceeds 10^25 floating-point operations.
- Rung 2, outside parties at the gate: in August 2024 the U.S. AI Safety Institute at NIST signed agreements giving it access to major new models from Anthropic and OpenAI before and after public release, and the joint U.S. and U.K. evaluation of OpenAI's o1, published in December 2024, shared findings with OpenAI before the model's release.
- Rung 3, non-public material needed: OpenAI's GDPval (2025) built 1,320 tasks with professionals averaging about 14 years of experience and open-sourced a 220-task gold subset, and its SWE-bench Verified post shows what happens to a public set once models have seen it.
- Rung 4, a paid commitment disclosed: none found. No model builder or evaluation supplier checked for this article has published the value and term of a purchase of held-out test material.
- Rung 5, refresh on record: refresh activity is visible in METR's expanded task suite and tau-bench's successor, but no source shows a refresh bought from an outside supplier.
Could release gates run without anyone buying evaluation data?
Yes, and that is the strongest alternative explanation. Every gate described above can be met with tests a lab writes itself, public benchmarks it downloads at no cost, or evaluations a government runs as a public function. Activity at the gate proves demand for testing, not demand for purchased data.
Parts of the public record lean that way. Anthropic's guide describes teams converting their own bug tracker and support queue into test cases, and invites product managers, customer success managers and salespeople to contribute tasks. GDPval's tasks were written and reviewed by experienced professionals for the benchmark itself, a route that needs no licensed archive. Pre-release testing under the NIST agreements is government work, not a customer contract.
Two observations would separate the explanations. If release gates create purchased demand, buyers should sign test-only licenses with refresh clauses tied to dated triggers, and sellers should report repeat purchases from the same buyer after a release or a contamination finding. If gates are mostly self-served, engineering posts and system cards will keep describing in-house suites while seller filings stay silent on evaluation.
How to measure release-gate demand with budget owners
Interview the people who decide whether a model ships and the people who sign for its tests, and code every answer by stage and dated cohort. No such study has been run or published by SourceX. The design is set out so an investor or a company can run it and compare results; results are not yet published.
At a glance
How to measure release-gate demand with budget owners
Checklist · 7 items
- Question: in each quarter, which release decisions relied on held-out tasks the team did not write itself, and were any of those tasks paid for?
- Population: evaluation leads, release or model-risk owners and procurement staff at model developers and at enterprises deploying agents, excluding anyone with a commercial tie to the interviewer.
- Cohorts: group organizations by the quarter of their first written request for outside evaluation data, so later conversions are credited to the right starting point.
- Stages: count written requests, sample evaluations, paid commitments and repeat purchases separately, each with its own denominator.
- Variables: trigger (release, model swap, rule, contamination or saturation), unit size in tasks, rights scope, any refresh clause, and whether an in-house or public alternative was tried first.
- Reporting: below 30 interviews, publish counts and ranges instead of percentages, and quote people only by role and date.
- Counter-case: report at least one team that replaced outside test material with an in-house suite, next to the headline result.
0 of 7 ticked. Tick each item as you go.
Where the evaluation-first thesis breaks
It breaks if buyers can meet release gates without paying outsiders, or if too few buyers face gates that need outside material. Each risk below comes with a marker an investor can check.
- In-house build: Anthropic's guide shows how much of a suite can come from a team's own failures and staff. If every buyer works that way, outside test material stays a niche.
- Free public tests: public benchmarks cost nothing, and buyers may accept their decay as long as successors keep appearing, as when OpenAI pointed developers elsewhere after retiring SWE-bench Verified.
- Narrow regulation: the EU evaluation duty covers general-purpose models with systemic risk, presumed above 10^25 training operations, a small group of providers able to build tests themselves.
- Public testers: government pre-release testing under agreements like NIST's may stand in for commercial evaluation at the frontier.
- Buyer concentration: on the training side, Reddit's annual report for 2024 places substantially all of its licensing contract value with two partners. Evaluation buyers could be as few, making revenue lumpy and dependent on renewals.
- Leakage after delivery: a held-out set loses value once models see it, so a seller whose sets leak, or who also sells training data to the same buyer, may find its tests discounted.
What to ask a seller that claims release-gate demand, SourceX included
Ask for evidence at the rung the seller claims, in terms you can check. Put the same questions to SourceX.
SourceX is one place to test this. It turns records a company selects into datasets licensed for AI and manages the whole deal: it sources and qualifies the supply, reviews rights, handles contracting, prepares and delivers the data, and handles payment. Datasets are found among U.S. businesses when a buyer asks rather than kept in stock, and the record owner must approve the scope and terms and sign before anything is delivered, which is the point where a testing-only scope would be written down.
SourceX has not published how its requests divide between evaluation and training uses; company and commercial metrics are something it provides to investors directly. The raise it is running under Rule 506(c) is for accredited investors, and a reader who requests materials through the investors page can bring any question below.
Checklist · 6 items
- Which rung of the release-gate ladder does your best evidence reach, and what dated document supports it?
- How many written requests named a release, model swap, rule or contamination finding as the trigger?
- What share of signed licenses permit testing only, and how is held-out status protected after delivery?
- Has any buyer bought refreshed test material, and which event prompted it?
- Who signs for evaluation data at your buyers, and is it the same person who signs for training data?
- Which of your buyers could meet the same gate with an in-house suite or a public benchmark?
0 of 6 ticked. Tick each item as you go.
Methods note: how the release-gate test was assembled
The release-cycle map, the six-test matrix, the five-rung ladder and the interview design are frameworks this article defines, built on 2026-10-26. Their evidence comes from Anthropic's January 2026 evaluation guide and its Responsible Scaling Policy page, both read directly; the tau-bench repository, read on GitHub; and pages verified through searches restricted to each publisher's domain: NIST, EUR-Lex, METR, OpenAI (GDPval, the SWE-bench Verified post and the data partnerships page) and Reddit's annual report for 2024 on SEC EDGAR.
Not measured: spending on evaluation or training data at any company, prices, how often buyers re-purchase test material, who signs for it, or any SourceX data. Rung placements record the highest public evidence found by the check date, not what labs do in private, and an empty rung means no public source was found, not that no purchase exists.
Frequently asked questions
Is evaluation data bought before training data?
Within a single release, the test usually comes first: a team needs it to measure the gap before it knows which training data would close it. Anthropic's January 2026 guide recommends building evaluations before an agent can pass them. Whether that test is bought from outside or written in-house is a separate question with little public evidence.
Why does evaluation data need refreshing more often than training data?
A test loses value as models see it, saturate it or outgrow it. OpenAI stopped evaluating SWE-bench Verified after finding contamination and flawed tests, and METR expanded its task suite from 170 to 228 tasks in January 2026. A training dataset is consumed in a training run, so its clock is set by model generations and contract terms instead.
What evidence would show a paid evaluation data market?
Paid commitments for held-out test material with a stated term, and repeat purchases tied to dated triggers such as a release or a contamination finding. Public evidence reaches the gates and the refresh activity but not the payments, so investors need seller contracts and buyer interviews coded by cohort.
Does regulation force developers to buy evaluation data?
No. The EU AI Act requires providers of general-purpose models with systemic risk to evaluate them with state-of-the-art protocols, including adversarial testing, but it does not say who must build the tests. Providers can meet the duty in-house, so regulation creates testing activity, not necessarily outside purchases.
Has SourceX published data on evaluation versus training requests?
No. Request, deal and commercial figures are not published; investors receive company and commercial metrics from SourceX directly. Materials can be requested at sourcex.si/investors, and any offering goes only to verified accredited investors, through SourceX's offering documents.
Sources
- Anthropic's guide to agent evaluations (January 9, 2026) recommends building evaluations before agents can pass them, calls 20 to 50 tasks from real failures a great start, says teams with evaluations can upgrade to new models in days rather than weeks, describes capability evaluations graduating into regression suites, and notes SWE-bench Verified scores rising from 30 percent toward saturation above 80 percent. Source
- Anthropic's Responsible Scaling Policy page lists versions from 1.0 (effective September 19, 2023) to 3.4 (effective July 8, 2026) and says the company will publicly release key information related to the evaluation and deployment of its models. Source
- In August 2024 the U.S. AI Safety Institute at NIST signed agreements with Anthropic and OpenAI that give it access to major new models before and after their public release. Source
- The joint U.S. and U.K. pre-deployment evaluation of OpenAI's o1 (December 2024) used a limited period of pre-deployment access and shared findings with OpenAI before the model was publicly released. Source
- The EU AI Act (Regulation (EU) 2024/1689) requires providers of general-purpose AI models with systemic risk to perform model evaluation with standardised, state-of-the-art protocols and tools, including documented adversarial testing, and presumes systemic risk above 10^25 floating-point operations of training compute. Source
- METR reported in March 2025 that the length of tasks AI agents can complete with 50 percent reliability has doubled roughly every seven months over six years. Source
- METR's Time Horizon 1.1 update (January 2026) expanded its task suite from 170 to 228 tasks and roughly doubled the number of tasks estimated at eight hours or longer. Source
- OpenAI says it no longer evaluates SWE-bench Verified: at least 59.4 percent of problems in an audited subset had flawed tests, models reproduced solution details from task identifiers, and progress slowed from 74.9 to 80.9 percent over six months. Source
- The tau-bench repository warns that its airline and retail tasks are not updated and points users to a successor benchmark with fixed tasks and a new banking domain. Source
- OpenAI's GDPval covers 44 occupations and 1,320 tasks written and reviewed by experienced professionals, with a 220-task gold subset open-sourced. Source
- OpenAI's data partnerships page seeks large-scale datasets that are not already easily accessible online to the public and says it is not seeking sensitive or personal information or information that belongs to a third party. Source
- Reddit's Form 10-K for fiscal 2024 says its content agreements are medium-term in length and that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners, with renewal risk. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.