AI data market
Evaluation dataset refresh revenue: re-runs are not repeat purchases
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Evaluation dataset refresh revenue is real only where decay consumes the test items themselves. Contamination, saturation and changes in the buyer's own work force replacement; new model releases, harness changes and regulatory testing mostly create re-runs on items the buyer already holds. LiveBench's public changelog shows both patterns. Underwrite repeat paid refreshes tied to dated triggers, not subscriptions or announcements, and check buyer concentration first.
Evaluation split
Evaluation dataset refresh
Key takeaways
- Separate the two demands: a refresh buys new test items, while a re-run buys compute and tooling on items the buyer already owns.
- Contamination, saturation and drift in the buyer's own workflows consume items; model releases consume them only when they push a set toward its ceiling.
- LiveBench replaced existing questions in 5 of its 11 changelog updates after launch, on an irregular schedule and for free, so refresh demand alone does not prove revenue.
- The strongest evidence is a second paid refresh from the same buyer tied to a dated trigger, counted by cohort and kept apart from requests and evaluations.
- Concentration, in-house builds, free public refreshes and re-run substitution are the risks to test before the upside.
Can evaluation refresh become recurring revenue?
It can, but only for evaluation sets whose value is used up by decay that requires new items, and only where the buyer cannot regenerate those items cheaply. That is a narrower thesis than recurring evaluation spending. A model developer that re-scores every release on the same private set spends again and again on compute, graders and engineers; a data supplier earns from that cycle only when the set itself has to be rebuilt.
The investor question is therefore not whether evaluation matters, which public evidence settles, but which part of evaluation spending recurs as a purchase of new data. This article separates refresh triggers from re-run triggers, tests the split against one public benchmark's own change history, sets out a ladder of demand evidence from announcement to second paid refresh, compares contract structures, and shows an explicitly illustrative model of how replacement share, cadence and retention combine.
Nothing here measures a market size or reports SourceX results. The evidence is public and dated, the framework is editorial, and the model's inputs are assumptions labeled as such.
Which triggers consume test items, and which only create re-runs?
Contamination, saturation, drift in the buyer's own work and discovered test defects consume items; new model releases, harness changes and regulatory evaluation duties mostly create re-runs. The distinction matters because only the first group gives a buyer a reason to pay for data again.
Contamination is the clearest consumer of items. Zhang and colleagues built GSM1k, 1,205 new grade-school math problems designed to mirror the public GSM8k benchmark, and the current version of their 2024 paper reports accuracy drops of up to 8 percent (the first preprint said 13) for some model families, notably Mistral and Phi, while most frontier models showed minimal signs of overfitting. In a post checked on October 26, 2026, OpenAI says it no longer evaluates SWE-bench Verified because the benchmark is increasingly contaminated and mismeasures frontier coding progress, and that an audit found at least 59.4 percent of the audited problems had flawed tests that reject functionally correct submissions.
Saturation consumes items more quietly. Anthropic's engineering guide to agent evaluations defines it as the point where an agent passes all of the solvable tasks, leaving no room for improvement, and distinguishes capability evaluations, which should start at low pass rates, from regression evaluations, which should sit near 100 percent. A saturated capability set still earns its keep as a regression suite, which means it keeps generating re-runs long after it stops generating purchases.
Re-run triggers are larger in volume and weaker for a data supplier. Under the European Commission's guidelines for general-purpose AI providers, models with systemic risk, presumed above 10 to the 25th floating-point operations of training compute, must be evaluated with standardised protocols and state-of-the-art tools, including documented adversarial testing. That duty recurs with every covered model, but nothing in it requires privately built or newly built items. The matrix sorts seven triggers by whether they consume items:
| Trigger | Consumes items? | Dated public evidence | What repeat procurement looks like | Disconfirming evidence |
|---|---|---|---|---|
| Contamination | Yes: leaked items stop measuring capability | GSM1k paper, 2024; OpenAI on SWE-bench Verified, checked October 2026 | A buyer replaces a slice after a leak or a suspicious score jump | Buyers keep leaked sets and discount the scores instead of replacing items |
| Saturation | Yes for capability sets; saturated sets become regression suites | Anthropic evals guide; LiveBench changelog, November 2024 and November 2025 | Harder items bought when top scores cluster near the ceiling | Buyers move to the next public benchmark rather than paying for private items |
| Drift in the buyer's own work | Yes for items tied to a policy, product or workflow | Editorial: depends on each buyer's change log | Items rebuilt after a policy or product change | The buyer's tasks change too slowly to need new items within a contract term |
| Test defects found | Partly: flawed items are repaired or dropped | OpenAI's SWE-bench Verified screening and later audit, checked October 2026 | Paid repair or replacement of flawed items | Buyers repair tests in-house without new data |
| New model release | Only when it pushes a set toward its ceiling | LiveBench, November 2024: tasks made harder after very high scores | A re-score on owned items; a purchase only if saturation follows | Releases move scores while the item set stays fit for use |
| Harness or grader change | No: same items, new runs | LiveBench, October 2025: every model re-run after a harness switch | Compute and engineering, not data | None needed: this trigger never buys items |
| Regulatory evaluation duties | No: protocols, not new items | European Commission GPAI guidelines, checked October 2026 | Testing services and documentation | Rules that later require private or fresh test data |
What LiveBench's changelog shows about refresh in practice
A benchmark designed around refresh replaced existing questions in 5 of its 11 updates after launch, named contamination or saturation in 4 of them, and did it on an irregular schedule rather than the monthly cadence it first announced. That is evidence that refresh demand is real and triggered, and also a reminder that the best-known refresh-by-design benchmark is free.
LiveBench launched on June 12, 2024 with 960 questions, saying it would release new question sets monthly to limit contamination and that every question has an objective ground-truth answer. From July 2024 it held the set at 1,000 questions, with each update adding and removing the same number. Its changelog then records 11 dated updates in the roughly 19 months through January 8, 2026. The README notes that not all questions in the current release are public, a sign that even an open benchmark protects its newest items.
Two entries show the split this article draws. In November 2024 the maintainers refreshed questions to check for contamination and made tasks harder because o1 and other reasoning models had reached very high scores: a model release consuming items through saturation. In October 2025 they switched the agent harness for agentic coding and re-ran every model on the existing tasks: a re-run that consumed no items at all.
The coding below reads only the reasons each changelog entry states; an entry can carry more than one reason, so the counts overlap:
| Reason stated in the entry | Entries | Count of 11 | What was consumed |
|---|---|---|---|
| Existing questions replaced or refreshed | Aug 2024, Nov 2024, Apr 2025 (two), Nov 2025 | 5 | Items, at a fixed set size |
| Contamination named | Nov 2024, Apr 2025, Nov 2025 | 3 | Items replaced with newer sources |
| Saturation or very high scores named | Jun 2024, Nov 2024, Nov 2025 | 3 | A task removed or made harder |
| New task or category added | Jul 2024, May 2025, Dec 2025, Jan 2026 | 4 | New coverage, not replacement |
| Harness change with a full re-run | Oct 2025 | 1 | Compute only |
The demand-signal ladder: from a public signal to a second paid refresh
Grade refresh evidence by how close it sits to a repeat payment, and count each rung by dated cohort so that requests, evaluations and paid commitments are never added together. Announcements and benchmark retirements show that decay happens; only the top rungs show that someone pays to fix it more than once.
At a glance
The demand-signal ladder: from a public signal to a second paid refresh
The method behind the ladder is the one this question calls for: interview the people who own evaluation budgets at independent buyers, log each request, evaluation and paid commitment with its date and stated trigger, and follow each cohort through at least one full refresh cycle before reporting a repeat rate. No results from such a study are published here, and SourceX has not published one.
Six rungs, weakest first:
- Rung 1, public decay signal: a benchmark retired, refreshed or flagged as contaminated. It proves the problem exists, not that anyone buys a fix.
- Rung 2, written request naming a trigger: a buyer asks for new items because of contamination, saturation or a change in its own work. Count requests per buyer, not per conversation.
- Rung 3, paid evaluation set with refresh terms: a first purchase whose contract defines how a refresh would be ordered and priced.
- Rung 4, first triggered repurchase: the buyer pays for replacement items after a dated trigger. This is the first rung that counts as refresh revenue.
- Rung 5, second paid refresh from the same buyer: repetition under the same or a renewed contract, the rung that separates a pattern from an event.
- Rung 6, contracted refresh schedule with anchor items: committed future refreshes, with a fixed anchor subset that keeps scores comparable and makes switching suppliers costly.
Which contract structures make a refresh repeat?
Structures that tie payment to a defined trigger or schedule, and that keep items private, make refreshes repeat; structures that sell a set once and leave decay to the buyer do not. The comparison below sets five structures against who carries the decay risk and what an investor should ask to see.
Public filings show that multi-year data licensing exists, and how concentrated it can be. Reddit's FY2024 Form 10-K reports $252.9 million of remaining performance obligations in contracts with an original expected duration of more than one year, consisting primarily of long-term content licensing contracts. The same filing states: "Although we have negotiated content agreements with a number of partners that are medium-term in length, to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners." That is content licensing rather than evaluation, but it shows what term contracts and buyer concentration look like once they reach a filing.
Construction cost explains why a buyer might pay rather than rebuild. OpenAI's GDPval, released in 2025, covers 44 occupations from the top nine industries contributing to U.S. GDP with 1,320 tasks crafted and vetted by experienced professionals, and its paper says the tasks take an expert an average of seven hours to complete. OpenAI open-sourced a gold subset of 220 of those tasks. Items that costly to build, and worth more while unpublished, are the kind a buyer might replace through a supplier rather than in-house.
Five structures compared:
| Structure | Who carries decay risk | Repeat revenue signal | Contamination control | Ask to see |
|---|---|---|---|---|
| One-time set license | Buyer | None until the buyer chooses to buy again | Depends on the buyer's handling | Whether any buyer bought a second set, and why |
| Triggered refresh order | Shared: the buyer decides, the supplier must deliver | Repurchases tied to dated triggers | Supplier keeps unreleased reserve items | A trigger log matched to each repurchase date |
| Pre-priced refresh option | Supplier holds capacity; buyer holds the choice | Exercise rate of options by cohort | As for triggered orders | Options granted versus exercised, by contract start date |
| Scheduled refresh with anchor items | Supplier, within the schedule | Committed refreshes and renewal history | Anchor subset fixed; rotating slices replaced | Anchor policy, slice size and renewal rate |
| Evaluation run as a service | Supplier: items never leave its control | Recurring runs rather than item sales | Strongest: items stay private | Share of revenue from runs versus new items |
An illustrative model: replacement share, cadence and retention
Refresh volume per year, as a share of the original build, equals the share of items replaced per refresh times the number of refreshes per year; over several years it is scaled by how many buyers stay. The model below uses assumed inputs to show which lever matters. It is not a forecast of any market or company.
Assume buyers who stay keep refreshing on the same pattern and that 70 percent of buyers are retained each year. Over three years, refresh volume then equals share times cadence times 2.19: one full year, plus 0.7 of a year, plus 0.49 of a year from the buyers still present. At a 10 percent slice twice a year, three years of refreshes add up to 44 percent of the original build; at a quarter of the set four times a year, they reach 219 percent. A set that is only ever re-run adds nothing, however often it is scored.
Two costs sit outside the formula. Anchor items kept for comparability are not rebuilt but must be revalidated, and every refreshed slice needs its own check that the test still passes known-good answers, because new items carry new defects. Both scale with cadence, so the fast-decay scenario is also the most expensive to serve.
Illustrative scenarios, all inputs assumed:
| Scenario | Share replaced per refresh | Refreshes per year | Refresh volume per year | Three years at 70 percent retention |
|---|---|---|---|---|
| Re-run only | 0 percent | Any | 0 percent of the build | 0 percent of the build |
| Slow decay | 10 percent | 2 | 20 percent of the build | 44 percent of the build |
| Moderate decay | 25 percent | 2 | 50 percent of the build | 110 percent of the build |
| Fast decay | 25 percent | 4 | 100 percent of the build | 219 percent of the build |
What could break the refresh thesis?
Five developments could keep evaluation refreshes from becoming repeat revenue, and each has a public marker an investor can watch. Rank them before weighing the upside; if a company cannot name the one that would do it the most damage, it has not tested its own model.
- Buyer concentration. If a few model developers account for most refresh spending, one lost buyer resets the revenue line. Reddit's FY2024 10-K, with two partners behind substantially all of its licensing contract value, warns that those agreements could fail to renew or come back on less favorable terms at renewal.
- Free public refresh. LiveBench refreshes questions and publishes most of them; if public benchmarks keep pace with contamination and saturation, buyers can wait for the next release instead of paying.
- In-house builds. Guidance such as Anthropic's engineering guide is written for teams building their own agent evaluations, and a buyer that can build and refresh items cheaply needs no supplier.
- Re-run substitution. If buyers answer decay by moving saturated sets into regression suites and re-scoring rather than replacing items, evaluation spending can grow while refresh purchases stay flat.
- Standardised protocols. The EU evaluation duty for systemic-risk models points to standardised protocols and state-of-the-art tools; if regulators and buyers converge on shared benchmarks, private refresh loses its premium.
Diligence questions for a company selling evaluation refreshes
Ask questions that separate refresh revenue from re-run revenue and repeat buyers from one-off ones, and put them to every company in the category, SourceX included. SourceX's model is to help companies license selected business records to AI developers as datasets, with SourceX doing the sourcing, the rights review, the contracting, the preparation, the delivery and the payment; for buyers it sources datasets from U.S. businesses on request, holding no stock. Accredited investors are the audience for its Rule 506(c) raise, and company and commercial metrics reach them directly through the request form on its investors page.
Questions that test the refresh thesis:
- Which triggers caused each repurchase in the last eight quarters, and where is the dated trigger log?
- What share of evaluation revenue came from new items, and what share from runs, grading or tooling on items the buyer already held?
- How many buyers reached a second paid refresh, out of how many first purchases, by cohort start date?
- How are requests, evaluations and paid commitments defined and counted, and are they ever added together?
- What anchor policy keeps scores comparable across refreshes, and who owns the anchor items?
- How are unreleased items protected, and what happened the last time an item set leaked?
- What share of refresh revenue came from the two largest buyers?
Methods note: how the triggers and the changelog were coded
The trigger matrix is an editorial framework built from the public sources cited in each row and checked on October 26, 2026. Triggers were sorted by one criterion: whether the decay they describe requires replacing test items or only re-scoring items a buyer already holds. Drift in a buyer's own work has no public dataset behind it and is marked editorial.
The LiveBench analysis reads changelog.md and the README in a clone of the LiveBench GitHub repository made on October 26, 2026. Each of the 11 dated entries after the June 12, 2024 launch was coded only by reasons the entry states; reasons a reader might infer, such as contamination behind a source swap that does not mention it, were not coded. The July 2024 entry added questions while growing the set to 1,000 and is coded as new coverage, not replacement. One benchmark is not a market, and LiveBench is a free academic project, so the counts describe refresh behavior, not willingness to pay.
Not measured: market size, prices or contract values for evaluation data, buyer retention, or any SourceX figure. The model's inputs are assumptions chosen to show sensitivity, and the demand-signal ladder is a measurement design with no interview or cohort results reported.
Frequently asked questions
Is evaluation data a recurring revenue business?
It can be, where decay forces buyers to replace test items and they pay a supplier rather than rebuild in-house. Contamination, saturation and changes in a buyer's own workflows create that need, while model releases and harness changes mostly create re-runs. The proof is a second paid refresh from the same buyer tied to a dated trigger.
What is the difference between re-running and refreshing an evaluation set?
A re-run scores a new model or configuration on items the buyer already holds, so it buys compute, grading and engineering time. A refresh replaces some of the items themselves because they leaked, saturated or no longer match the task. Only refreshes create repeat demand for new data.
Why do AI benchmarks need refreshing?
Items leak into training data, top models reach the ceiling, and tests turn out to contain flaws. OpenAI stopped evaluating SWE-bench Verified over contamination and flawed tests, and LiveBench's changelog records refreshes for contamination and saturation between 2024 and 2025.
Does a subscription contract prove recurring evaluation demand?
Not on its own. A subscription shows committed payments, but an investor should check what each payment buys: new items, or runs and tooling on existing ones. Renewal history by cohort and a trigger log matched to refresh deliveries are stronger evidence than the contract label.
How should investors verify evaluation refresh revenue claims?
Ask for repurchases by cohort with their dated triggers, the split between item revenue and run revenue, the number of buyers that reached a second paid refresh, and the share of revenue from the two largest buyers. Requests, evaluations and paid commitments should be counted separately, never added together.
Sources
- OpenAI says it no longer evaluates SWE-bench Verified because it is increasingly contaminated and mismeasures frontier coding progress; an audit found at least 59.4 percent of audited problems had flawed tests that reject functionally correct submissions. Source
- OpenAI's SWE-bench Verified announcement describes professional developers screening SWE-bench samples because some tests were overly specific or unrelated and could reject correct solutions; 500 samples were verified. Source
- OpenAI's GDPval spans 44 occupations from the top nine industries contributing to U.S. GDP with 1,320 tasks crafted and vetted by experienced professionals, and a gold subset of 220 tasks is open-sourced. Source
- The GDPval paper says its tasks require an average of seven hours of work for an expert professional to complete. Source
- Zhang et al. (2024) built GSM1k, 1,205 problems mirroring GSM8k, and in the current version report accuracy drops of up to 8 percent for some model families, notably Mistral and Phi, with minimal overfitting signs for most frontier models. Source
- LiveBench's changelog records its June 12, 2024 launch (960 questions, monthly releases planned, objective answers), a fixed 1,000-question set from July 2024, and 11 dated updates through January 8, 2026 citing contamination, saturation, new tasks and a harness re-run. Source
- LiveBench's README says it limits contamination by releasing new questions monthly and that not all questions in the current release are public. Source
- Anthropic's guide defines eval saturation as an agent passing all solvable tasks, and says capability evals should start at low pass rates while regression evals should sit near 100 percent. Source
- Reddit's FY2024 Form 10-K reports $252.9 million of remaining performance obligations in contracts longer than one year, primarily long-term content licensing, and says substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
- The European Commission's GPAI guidelines say providers of models with systemic risk, presumed above 10^25 FLOP, must evaluate models with standardised protocols and state-of-the-art tools, including conducting and documenting adversarial testing. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.