Skip to content

AI data market

Evaluation dataset refresh revenue: re-runs are not repeat purchases

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Evaluation dataset refresh revenue is real only where decay consumes the test items themselves. Contamination, saturation and changes in the buyer's own work force replacement; new model releases, harness changes and regulatory testing mostly create re-runs on items the buyer already holds. LiveBench's public changelog shows both patterns. Underwrite repeat paid refreshes tied to dated triggers, not subscriptions or announcements, and check buyer concentration first.

Evaluation split

Evaluation dataset refresh

This page covers evaluation dataset refresh revenue: re-runs are not repeat purchases.Illustrative

Key takeaways

  • Separate the two demands: a refresh buys new test items, while a re-run buys compute and tooling on items the buyer already owns.
  • Contamination, saturation and drift in the buyer's own workflows consume items; model releases consume them only when they push a set toward its ceiling.
  • LiveBench replaced existing questions in 5 of its 11 changelog updates after launch, on an irregular schedule and for free, so refresh demand alone does not prove revenue.
  • The strongest evidence is a second paid refresh from the same buyer tied to a dated trigger, counted by cohort and kept apart from requests and evaluations.
  • Concentration, in-house builds, free public refreshes and re-run substitution are the risks to test before the upside.

Can evaluation refresh become recurring revenue?

It can, but only for evaluation sets whose value is used up by decay that requires new items, and only where the buyer cannot regenerate those items cheaply. That is a narrower thesis than recurring evaluation spending. A model developer that re-scores every release on the same private set spends again and again on compute, graders and engineers; a data supplier earns from that cycle only when the set itself has to be rebuilt.

The investor question is therefore not whether evaluation matters, which public evidence settles, but which part of evaluation spending recurs as a purchase of new data. This article separates refresh triggers from re-run triggers, tests the split against one public benchmark's own change history, sets out a ladder of demand evidence from announcement to second paid refresh, compares contract structures, and shows an explicitly illustrative model of how replacement share, cadence and retention combine.

Nothing here measures a market size or reports SourceX results. The evidence is public and dated, the framework is editorial, and the model's inputs are assumptions labeled as such.

Which triggers consume test items, and which only create re-runs?

Contamination, saturation, drift in the buyer's own work and discovered test defects consume items; new model releases, harness changes and regulatory evaluation duties mostly create re-runs. The distinction matters because only the first group gives a buyer a reason to pay for data again.

Contamination is the clearest consumer of items. Zhang and colleagues built GSM1k, 1,205 new grade-school math problems designed to mirror the public GSM8k benchmark, and the current version of their 2024 paper reports accuracy drops of up to 8 percent (the first preprint said 13) for some model families, notably Mistral and Phi, while most frontier models showed minimal signs of overfitting. In a post checked on October 26, 2026, OpenAI says it no longer evaluates SWE-bench Verified because the benchmark is increasingly contaminated and mismeasures frontier coding progress, and that an audit found at least 59.4 percent of the audited problems had flawed tests that reject functionally correct submissions.

Saturation consumes items more quietly. Anthropic's engineering guide to agent evaluations defines it as the point where an agent passes all of the solvable tasks, leaving no room for improvement, and distinguishes capability evaluations, which should start at low pass rates, from regression evaluations, which should sit near 100 percent. A saturated capability set still earns its keep as a regression suite, which means it keeps generating re-runs long after it stops generating purchases.

Re-run triggers are larger in volume and weaker for a data supplier. Under the European Commission's guidelines for general-purpose AI providers, models with systemic risk, presumed above 10 to the 25th floating-point operations of training compute, must be evaluated with standardised protocols and state-of-the-art tools, including documented adversarial testing. That duty recurs with every covered model, but nothing in it requires privately built or newly built items. The matrix sorts seven triggers by whether they consume items:

Which triggers consume test items, and which only create re-runs?
TriggerConsumes items?Dated public evidenceWhat repeat procurement looks likeDisconfirming evidence
ContaminationYes: leaked items stop measuring capabilityGSM1k paper, 2024; OpenAI on SWE-bench Verified, checked October 2026A buyer replaces a slice after a leak or a suspicious score jumpBuyers keep leaked sets and discount the scores instead of replacing items
SaturationYes for capability sets; saturated sets become regression suitesAnthropic evals guide; LiveBench changelog, November 2024 and November 2025Harder items bought when top scores cluster near the ceilingBuyers move to the next public benchmark rather than paying for private items
Drift in the buyer's own workYes for items tied to a policy, product or workflowEditorial: depends on each buyer's change logItems rebuilt after a policy or product changeThe buyer's tasks change too slowly to need new items within a contract term
Test defects foundPartly: flawed items are repaired or droppedOpenAI's SWE-bench Verified screening and later audit, checked October 2026Paid repair or replacement of flawed itemsBuyers repair tests in-house without new data
New model releaseOnly when it pushes a set toward its ceilingLiveBench, November 2024: tasks made harder after very high scoresA re-score on owned items; a purchase only if saturation followsReleases move scores while the item set stays fit for use
Harness or grader changeNo: same items, new runsLiveBench, October 2025: every model re-run after a harness switchCompute and engineering, not dataNone needed: this trigger never buys items
Regulatory evaluation dutiesNo: protocols, not new itemsEuropean Commission GPAI guidelines, checked October 2026Testing services and documentationRules that later require private or fresh test data
Seven evaluation triggers sorted into two outcomes: contamination, saturation, drift in the buyer's work and test defects lead to replacing test items, while model releases, harness changes and regulatory duties lead to re-runs on items the buyer already holds, with model releases also linked to replacement through saturation.
Refresh triggers versus re-run triggers. Editorial classification from the sources cited in the matrix.

What LiveBench's changelog shows about refresh in practice

A benchmark designed around refresh replaced existing questions in 5 of its 11 updates after launch, named contamination or saturation in 4 of them, and did it on an irregular schedule rather than the monthly cadence it first announced. That is evidence that refresh demand is real and triggered, and also a reminder that the best-known refresh-by-design benchmark is free.

LiveBench launched on June 12, 2024 with 960 questions, saying it would release new question sets monthly to limit contamination and that every question has an objective ground-truth answer. From July 2024 it held the set at 1,000 questions, with each update adding and removing the same number. Its changelog then records 11 dated updates in the roughly 19 months through January 8, 2026. The README notes that not all questions in the current release are public, a sign that even an open benchmark protects its newest items.

Two entries show the split this article draws. In November 2024 the maintainers refreshed questions to check for contamination and made tasks harder because o1 and other reasoning models had reached very high scores: a model release consuming items through saturation. In October 2025 they switched the agent harness for agentic coding and re-ran every model on the existing tasks: a re-run that consumed no items at all.

The coding below reads only the reasons each changelog entry states; an entry can carry more than one reason, so the counts overlap:

What LiveBench's changelog shows about refresh in practice
Reason stated in the entryEntriesCount of 11What was consumed
Existing questions replaced or refreshedAug 2024, Nov 2024, Apr 2025 (two), Nov 20255Items, at a fixed set size
Contamination namedNov 2024, Apr 2025, Nov 20253Items replaced with newer sources
Saturation or very high scores namedJun 2024, Nov 2024, Nov 20253A task removed or made harder
New task or category addedJul 2024, May 2025, Dec 2025, Jan 20264New coverage, not replacement
Harness change with a full re-runOct 20251Compute only

The demand-signal ladder: from a public signal to a second paid refresh

Grade refresh evidence by how close it sits to a repeat payment, and count each rung by dated cohort so that requests, evaluations and paid commitments are never added together. Announcements and benchmark retirements show that decay happens; only the top rungs show that someone pays to fix it more than once.

At a glance

The demand-signal ladder: from a public signal to a second paid refresh

The method behind the ladder is the one this question calls for: interview the people who own evaluation budgets at independent buyers, log each request, evaluation and paid commitment with its date and stated trigger, and follow each cohort through at least one full refresh cycle before reporting a repeat rate. No results from such a study are published here, and SourceX has not published one.

Six rungs, weakest first:

  • Rung 1, public decay signal: a benchmark retired, refreshed or flagged as contaminated. It proves the problem exists, not that anyone buys a fix.
  • Rung 2, written request naming a trigger: a buyer asks for new items because of contamination, saturation or a change in its own work. Count requests per buyer, not per conversation.
  • Rung 3, paid evaluation set with refresh terms: a first purchase whose contract defines how a refresh would be ordered and priced.
  • Rung 4, first triggered repurchase: the buyer pays for replacement items after a dated trigger. This is the first rung that counts as refresh revenue.
  • Rung 5, second paid refresh from the same buyer: repetition under the same or a renewed contract, the rung that separates a pattern from an event.
  • Rung 6, contracted refresh schedule with anchor items: committed future refreshes, with a fixed anchor subset that keeps scores comparable and makes switching suppliers costly.

Which contract structures make a refresh repeat?

Structures that tie payment to a defined trigger or schedule, and that keep items private, make refreshes repeat; structures that sell a set once and leave decay to the buyer do not. The comparison below sets five structures against who carries the decay risk and what an investor should ask to see.

Public filings show that multi-year data licensing exists, and how concentrated it can be. Reddit's FY2024 Form 10-K reports $252.9 million of remaining performance obligations in contracts with an original expected duration of more than one year, consisting primarily of long-term content licensing contracts. The same filing states: "Although we have negotiated content agreements with a number of partners that are medium-term in length, to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners." That is content licensing rather than evaluation, but it shows what term contracts and buyer concentration look like once they reach a filing.

Construction cost explains why a buyer might pay rather than rebuild. OpenAI's GDPval, released in 2025, covers 44 occupations from the top nine industries contributing to U.S. GDP with 1,320 tasks crafted and vetted by experienced professionals, and its paper says the tasks take an expert an average of seven hours to complete. OpenAI open-sourced a gold subset of 220 of those tasks. Items that costly to build, and worth more while unpublished, are the kind a buyer might replace through a supplier rather than in-house.

Five structures compared:

Which contract structures make a refresh repeat?
StructureWho carries decay riskRepeat revenue signalContamination controlAsk to see
One-time set licenseBuyerNone until the buyer chooses to buy againDepends on the buyer's handlingWhether any buyer bought a second set, and why
Triggered refresh orderShared: the buyer decides, the supplier must deliverRepurchases tied to dated triggersSupplier keeps unreleased reserve itemsA trigger log matched to each repurchase date
Pre-priced refresh optionSupplier holds capacity; buyer holds the choiceExercise rate of options by cohortAs for triggered ordersOptions granted versus exercised, by contract start date
Scheduled refresh with anchor itemsSupplier, within the scheduleCommitted refreshes and renewal historyAnchor subset fixed; rotating slices replacedAnchor policy, slice size and renewal rate
Evaluation run as a serviceSupplier: items never leave its controlRecurring runs rather than item salesStrongest: items stay privateShare of revenue from runs versus new items

An illustrative model: replacement share, cadence and retention

Refresh volume per year, as a share of the original build, equals the share of items replaced per refresh times the number of refreshes per year; over several years it is scaled by how many buyers stay. The model below uses assumed inputs to show which lever matters. It is not a forecast of any market or company.

Assume buyers who stay keep refreshing on the same pattern and that 70 percent of buyers are retained each year. Over three years, refresh volume then equals share times cadence times 2.19: one full year, plus 0.7 of a year, plus 0.49 of a year from the buyers still present. At a 10 percent slice twice a year, three years of refreshes add up to 44 percent of the original build; at a quarter of the set four times a year, they reach 219 percent. A set that is only ever re-run adds nothing, however often it is scored.

Two costs sit outside the formula. Anchor items kept for comparability are not rebuilt but must be revalidated, and every refreshed slice needs its own check that the test still passes known-good answers, because new items carry new defects. Both scale with cadence, so the fast-decay scenario is also the most expensive to serve.

Illustrative scenarios, all inputs assumed:

An illustrative model: replacement share, cadence and retention
ScenarioShare replaced per refreshRefreshes per yearRefresh volume per yearThree years at 70 percent retention
Re-run only0 percentAny0 percent of the build0 percent of the build
Slow decay10 percent220 percent of the build44 percent of the build
Moderate decay25 percent250 percent of the build110 percent of the build
Fast decay25 percent4100 percent of the build219 percent of the build

What could break the refresh thesis?

Five developments could keep evaluation refreshes from becoming repeat revenue, and each has a public marker an investor can watch. Rank them before weighing the upside; if a company cannot name the one that would do it the most damage, it has not tested its own model.

  • Buyer concentration. If a few model developers account for most refresh spending, one lost buyer resets the revenue line. Reddit's FY2024 10-K, with two partners behind substantially all of its licensing contract value, warns that those agreements could fail to renew or come back on less favorable terms at renewal.
  • Free public refresh. LiveBench refreshes questions and publishes most of them; if public benchmarks keep pace with contamination and saturation, buyers can wait for the next release instead of paying.
  • In-house builds. Guidance such as Anthropic's engineering guide is written for teams building their own agent evaluations, and a buyer that can build and refresh items cheaply needs no supplier.
  • Re-run substitution. If buyers answer decay by moving saturated sets into regression suites and re-scoring rather than replacing items, evaluation spending can grow while refresh purchases stay flat.
  • Standardised protocols. The EU evaluation duty for systemic-risk models points to standardised protocols and state-of-the-art tools; if regulators and buyers converge on shared benchmarks, private refresh loses its premium.

Diligence questions for a company selling evaluation refreshes

Ask questions that separate refresh revenue from re-run revenue and repeat buyers from one-off ones, and put them to every company in the category, SourceX included. SourceX's model is to help companies license selected business records to AI developers as datasets, with SourceX doing the sourcing, the rights review, the contracting, the preparation, the delivery and the payment; for buyers it sources datasets from U.S. businesses on request, holding no stock. Accredited investors are the audience for its Rule 506(c) raise, and company and commercial metrics reach them directly through the request form on its investors page.

Questions that test the refresh thesis:

  • Which triggers caused each repurchase in the last eight quarters, and where is the dated trigger log?
  • What share of evaluation revenue came from new items, and what share from runs, grading or tooling on items the buyer already held?
  • How many buyers reached a second paid refresh, out of how many first purchases, by cohort start date?
  • How are requests, evaluations and paid commitments defined and counted, and are they ever added together?
  • What anchor policy keeps scores comparable across refreshes, and who owns the anchor items?
  • How are unreleased items protected, and what happened the last time an item set leaked?
  • What share of refresh revenue came from the two largest buyers?

Methods note: how the triggers and the changelog were coded

The trigger matrix is an editorial framework built from the public sources cited in each row and checked on October 26, 2026. Triggers were sorted by one criterion: whether the decay they describe requires replacing test items or only re-scoring items a buyer already holds. Drift in a buyer's own work has no public dataset behind it and is marked editorial.

The LiveBench analysis reads changelog.md and the README in a clone of the LiveBench GitHub repository made on October 26, 2026. Each of the 11 dated entries after the June 12, 2024 launch was coded only by reasons the entry states; reasons a reader might infer, such as contamination behind a source swap that does not mention it, were not coded. The July 2024 entry added questions while growing the set to 1,000 and is coded as new coverage, not replacement. One benchmark is not a market, and LiveBench is a free academic project, so the counts describe refresh behavior, not willingness to pay.

Not measured: market size, prices or contract values for evaluation data, buyer retention, or any SourceX figure. The model's inputs are assumptions chosen to show sensitivity, and the demand-signal ladder is a measurement design with no interview or cohort results reported.

Frequently asked questions

Is evaluation data a recurring revenue business?

It can be, where decay forces buyers to replace test items and they pay a supplier rather than rebuild in-house. Contamination, saturation and changes in a buyer's own workflows create that need, while model releases and harness changes mostly create re-runs. The proof is a second paid refresh from the same buyer tied to a dated trigger.

What is the difference between re-running and refreshing an evaluation set?

A re-run scores a new model or configuration on items the buyer already holds, so it buys compute, grading and engineering time. A refresh replaces some of the items themselves because they leaked, saturated or no longer match the task. Only refreshes create repeat demand for new data.

Why do AI benchmarks need refreshing?

Items leak into training data, top models reach the ceiling, and tests turn out to contain flaws. OpenAI stopped evaluating SWE-bench Verified over contamination and flawed tests, and LiveBench's changelog records refreshes for contamination and saturation between 2024 and 2025.

Does a subscription contract prove recurring evaluation demand?

Not on its own. A subscription shows committed payments, but an investor should check what each payment buys: new items, or runs and tooling on existing ones. Renewal history by cohort and a trigger log matched to refresh deliveries are stronger evidence than the contract label.

How should investors verify evaluation refresh revenue claims?

Ask for repurchases by cohort with their dated triggers, the split between item revenue and run revenue, the number of buyers that reached a second paid refresh, and the share of revenue from the two largest buyers. Requests, evaluations and paid commitments should be counted separately, never added together.

Sources

  • OpenAI says it no longer evaluates SWE-bench Verified because it is increasingly contaminated and mismeasures frontier coding progress; an audit found at least 59.4 percent of audited problems had flawed tests that reject functionally correct submissions. Source
  • OpenAI's SWE-bench Verified announcement describes professional developers screening SWE-bench samples because some tests were overly specific or unrelated and could reject correct solutions; 500 samples were verified. Source
  • OpenAI's GDPval spans 44 occupations from the top nine industries contributing to U.S. GDP with 1,320 tasks crafted and vetted by experienced professionals, and a gold subset of 220 tasks is open-sourced. Source
  • The GDPval paper says its tasks require an average of seven hours of work for an expert professional to complete. Source
  • Zhang et al. (2024) built GSM1k, 1,205 problems mirroring GSM8k, and in the current version report accuracy drops of up to 8 percent for some model families, notably Mistral and Phi, with minimal overfitting signs for most frontier models. Source
  • LiveBench's changelog records its June 12, 2024 launch (960 questions, monthly releases planned, objective answers), a fixed 1,000-question set from July 2024, and 11 dated updates through January 8, 2026 citing contamination, saturation, new tasks and a harness re-run. Source
  • LiveBench's README says it limits contamination by releasing new questions monthly and that not all questions in the current release are public. Source
  • Anthropic's guide defines eval saturation as an agent passing all solvable tasks, and says capability evals should start at low pass rates while regression evals should sit near 100 percent. Source
  • Reddit's FY2024 Form 10-K reports $252.9 million of remaining performance obligations in contracts longer than one year, primarily long-term content licensing, and says substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source
  • The European Commission's GPAI guidelines say providers of models with systemic risk, presumed above 10^25 FLOP, must evaluate models with standardised protocols and state-of-the-art tools, including conducting and documenting adversarial testing. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies