Skip to content

AI data market

Synthetic evaluation dataset risks: three tests training rules skip

By SourceX Editorial · Reviewed by Noah Loul ·

8 sections

Short answer

Synthetic data can carry much of a model's training, but an evaluation set is a measuring instrument, and synthetic evaluation dataset risks come in three forms training augmentation can absorb: items may not resemble deployment (realism), may share a lineage with the models they score (independence), or may already be known to them (contamination). Investors should ask for each test separately, plus a matched-budget mixture study against an untouched real set.

Evaluation split

Synthetic evaluation dataset

This page covers synthetic evaluation dataset risks: three tests training rules skip.Illustrative

Key takeaways

  • Training-side substitution is real: NVIDIA reports that more than 98 percent of Nemotron-4 340B's alignment data was synthetic.
  • Evaluation errors become the score, so synthetic test items need proofs that synthetic training data does not.
  • Realism, independence and contamination fail separately and need separate tests, including contamination during the test itself.
  • Synthetic retrieval test collections have been reported to evaluate reliably, which is the strongest case against the thesis.
  • A controlled-mixture experiment with a real split-half ceiling would price how much synthetic share an evaluation can carry; no results exist yet.

Why can't an evaluation set borrow the training-data policy?

Because an evaluation set is the measuring instrument, and nothing downstream checks the instrument. When synthetic training data is wrong, a held-out test can catch the damage. When synthetic evaluation data is wrong, the error becomes the score, and every decision built on that score inherits it.

Substitution on the training side is already large. NVIDIA's Nemotron-4 340B technical report (June 2024) says more than 98 percent of the data used in its model alignment process was synthetically generated. A policy that accepts synthetic data whenever it improves results on a test works well there, because the test is the backstop.

The same policy cannot govern the test itself. That gives investors a thesis they can check: if evaluation keeps a stricter, separate policy, demand for real held-out records survives even as synthetic data absorbs training budgets; if synthetic evaluation proves itself property by property, that demand shrinks. What follows sets out the three properties, where each one binds, the experiment that would settle the question and the risks to the thesis.

Synthetic evaluation dataset risks: three properties that fail separately

Treat them as three tests, because a synthetic evaluation set can pass one and fail another. Realism asks whether the items resemble the situations the system will meet. Independence asks whether the items, or the grader, share a lineage with the models being scored. Contamination asks whether the models could already have seen the items or their answers.

Independence has the clearest experimental evidence. Panickssery, Bowman and Feng (NeurIPS 2024) found that models such as GPT-4 and Llama 2 can tell their own outputs from those of other models and humans with non-trivial accuracy, and that shifting self-recognition through fine-tuning moved self-preference along a linear correlation when the model acted as an evaluator. A follow-up by Chen and colleagues (2025) adds a qualification that matters: stronger models show more self-preference, and much of it lines up with objectively better performance.

Contamination does not stop at training data. The team behind Humanity's Last Exam released a large public question set while keeping a private test set to assess overfitting, and a 2025 study of search-time contamination found that for about 3 percent of that benchmark's questions, search-based agents could directly find datasets with the ground-truth labels on Hugging Face. LiveBench answers the same threat from the other side: its maintainers say it limits contamination by releasing new questions monthly, and its README notes that not all questions in the current release are public.

Realism is where regulation speaks most directly. For high-risk AI systems, Article 10(3) of the EU AI Act sets quality criteria for the data used to train, validate and test them: the sets must suit the intended purpose, represent it adequately, and carry as few errors and gaps as can be achieved. The duty names testing data explicitly, and it applies only to systems in the high-risk category.

How each property fails, and the test an investor should ask a company to show:

Synthetic evaluation dataset risks: three properties that fail separately
PropertyQuestion it answersHow a synthetic set fails itTest to request
RealismDo items resemble deployment cases?Clean, single-intent items that skip the messy casesCompare item features with a sample of real cases
Independence of the item writerDid a candidate's model family write the items?Items phrased in a style one family favorsScore candidates by family and report the generator family's gap
Independence of the graderDoes a candidate's model family grade the answers?Self-preference lifts that family's scoresSwap the judge model and report rank changes
Contamination through trainingCould candidates have trained on the items or their seeds?Items generated from public documents the models readGenerate from sealed private seeds and compare with public seeds
Contamination at test timeCan an agent look up answers while being tested?Published items with labels are searchableKeep a private split; block or log retrieval during runs

The substitution matrix: where synthetic items can take budget

Synthetic items can take most of the budget where errors are caught downstream or a rule checks the answer, and very little where a score drives an outside decision. The matrix rates seven data jobs on the three properties and gives an editorial verdict for each; the verdicts are judgments drawn from the evidence above, not measurements.

The pattern is what matters for a market thesis. The top three rows are where generation competes head-on with licensed data, and NVIDIA's alignment figure shows how far that has already gone. The bottom three rows are where an evaluation's value depends on independence from the systems it scores and on resembling the setting they serve, which is the part of the market most open to real, held-out business records.

The substitution matrix, dated 2026-10-27:

The substitution matrix: where synthetic items can take budget
Data jobRealism needIndependence needContamination exposureSynthetic role
Pretraining and instruction augmentationModerateLowLowSubstitute, policed by held-out tests
Preference and reward dataModerateModerateLowSubstitute with human calibration
Development smoke testsLowLowModerateSubstitute
Regression suitesModerateModerateHigh once publishedComplement a real anchor slice
Model selection and release gatesHighHighHighReal core; synthetic probes reported apart
Procurement acceptance testsHighHighHighReal core from the buyer's own setting
Testing for high-risk systems under the EU AI ActVery highHighHighReal core; synthetic only as a documented supplement

What the strongest case for synthetic evaluation gets right

Synthetic evaluation works where its truth can be checked and its rankings have been tested against real data, and an investor should not wave that away. The policy argued for here is a set of proofs, one per property, not a ban.

The best evidence comes from search. Rahmani and colleagues (SIGIR 2024) used language models to generate both the queries and the relevance judgments for a test collection and reported that such synthetic collections can reliably be used for retrieval evaluation; they also examined the risk that synthetic collections favor systems built on language models. That is what a convincing claim looks like: synthetic items, a real collection to compare against and an explicit check for the independence failure.

Two other results narrow the gap. LiveBench scores each question against verifiable, objective ground-truth answers without an LLM judge, which removes grader self-preference for those task types. And Chen and colleagues' finding that much of strong models' self-preference reflects genuinely better answers suggests the independence penalty may be smaller for frontier graders than early studies implied.

What these results do not show is transfer to business tasks whose correct answer depends on a company's own policies, records and outcomes. A retrieval collection over public passages has a real comparison set to hand; a claims-handling or dispatch evaluation usually does not, unless someone supplies untouched real cases. That gap is what the experiment below is built to price.

The controlled-mixture experiment: how much synthetic can a test absorb?

Run one panel of models against evaluation sets that differ only in their synthetic share, and measure how far each set's ranking drifts from a sealed real set. The question is not whether synthetic items are good in general but how much synthetic share a particular evaluation can carry before it changes the decisions it exists to make.

Design: fix a panel of at least eight candidate models from at least three model families, including the generator's family. Seal a real reference set that the generator never sees, that is never published and that no candidate is tuned on. Build five evaluation sets with 0, 25, 50, 75 and 100 percent synthetic items, matched on item count, task mix and expert review hours, then score every candidate on every set and on the sealed reference.

Metrics: rank agreement with the sealed reference, measured with Kendall's tau, agreement on the top candidate, and the number of pass-or-fail calls that flip at the release threshold. Use agreement between two random halves of the real reference as the ceiling: a synthetic share is acceptable only if its agreement with the full reference cannot be distinguished statistically from that real split-half agreement. Report the generator family's score gap separately, and repeat the scoring with the judge swapped to another family.

This is a method, and its results are not yet published: the protocol was drawn up for this article, and no figures from it exist. The five-step grid and the eight-model panel are design choices a team can change. The split-half ceiling is the part to keep, because it ties the acceptance rule to how consistent real data is with itself rather than to a threshold picked in advance.

The experiment plan, component by component:

The controlled-mixture experiment: how much synthetic can a test absorb?
ComponentWhat it containsWhat it isolatesReported as
Sealed real referenceReal cases never seen by the generator or publishedThe ranking the decision should followModel ranking and pass-or-fail calls
Real split-halfTwo random halves of the referenceHow consistent real data is with itselfAgreement ceiling with an interval
Synthetic share armsSets at 0, 25, 50, 75 and 100 percent syntheticDrift caused by synthetic share aloneKendall's tau, top-candidate match, decision flips
Generator-family sliceCandidates from the generator's familyIndependence of the item writerScore gap against other families
Judge swapThe same answers graded by a second judge familyIndependence of the graderRank changes after the swap
Seed provenance splitItems from public versus sealed private seedsContamination through generationScore gap by seed source
A two-column comparison of the rules for synthetic training data and for evaluation sets across realism, independence and contamination, above the five synthetic-share arms of the mixture experiment scored against a sealed real set.
Same data, stricter rules for evaluation. Editorial framework; the mixture arms are a design, not results.

Risks: five ways the evaluation-policy thesis could fail

The thesis fails if any of five developments plays out, and each deserves a place in the model before the upside does:

  • Synthetic collections keep passing the test. If results like Rahmani and colleagues' retrieval findings extend to business tasks, rankings from synthetic sets will match real ones and buyers will move evaluation budgets to generation.
  • Deployers use their own production data. A company running an AI system can build evaluation cases from its own logged traffic and outcomes without licensing anything; the thesis holds mainly where a buyer lacks such records or cannot use them for evaluation.
  • Real sets wear out too. Once a real evaluation set is used and shared it can leak like any other, as the search-time contamination study showed for about 3 percent of one benchmark's questions. Real records hold their value only while they stay sealed, which favors refresh cycles over one-off sales.
  • Regulation stays narrow. Article 10 of the EU AI Act covers high-risk systems, and many business deployments may sit outside that category, leaving realism requirements to buyers' own procurement rules.
  • Graders improve. If frontier judges' self-preference mostly tracks real quality, as Chen and colleagues suggest, the independence argument for real data weakens and realism and contamination must carry the thesis alone.

Diligence questions for evaluation-data sellers, SourceX included

Ask any company that sells evaluation data, or that claims synthetic evaluation replaces it, for evidence on each property separately, using definitions you set rather than ones it offers. SourceX belongs on the same list.

SourceX is a place to apply these questions, not evidence for the thesis. It converts business records that companies choose to license into AI datasets and runs the chain from sourcing and qualification through rights review, contracting, preparation, delivery and payment, finding each dataset at U.S. businesses when a buyer asks rather than holding stock. Its deal, revenue and commercial figures are not public: the company gives company and commercial metrics to investors directly, and it is raising from accredited investors under Rule 506(c), with investor materials available on request through its investor page.

Questions that keep the three properties apart:

  • For each evaluation set you sell or use, who generated the items, which model families graded them, and which candidates share either lineage?
  • How are real held-out cases kept sealed, and how would you detect that one had been published or used for tuning?
  • What share of each evaluation set is synthetic, and has that share been tested against a sealed real reference?
  • Which of your buyers evaluate high-risk systems under the EU AI Act, and which rely on their own production logs instead?
  • What refresh cycle keeps a real evaluation set from wearing out, and who pays for each refresh?
  • Which result would persuade you that synthetic evaluation can replace real cases in a category you supply?

Methods note: sources and limits of the matrix and experiment plan

The three-property framework, the substitution matrix and the controlled-mixture plan were written for this article on 2026-10-27 as editorial tools. They draw on NVIDIA's Nemotron-4 340B report, the self-preference studies by Panickssery and colleagues and by Chen and colleagues, Rahmani and colleagues' synthetic test collections paper, the LiveBench repository, the Humanity's Last Exam paper, the 2025 search-time contamination study and Regulation (EU) 2024/1689. Matrix ratings are judgments from that evidence, not scores.

Not measured: the share of any company's evaluation budget spent on synthetic or real items, how often synthetic and real rankings agree on business tasks, the size of generator-family gaps outside the cited studies, and any SourceX metric. The experiment plan is a protocol, and no results from it exist.

Frequently asked questions

Can synthetic data be used in an evaluation set at all?

Yes, with conditions. Synthetic items suit development smoke tests, robustness probes and tasks whose answers a rule can check, and synthetic retrieval collections have been reported to support reliable evaluation. Decision-grade evaluations should still anchor on real held-out cases and report synthetic items as a separate slice.

What does independence mean in AI evaluation?

It means the items and the grader do not share a lineage with the models being scored. Research has found that language models can recognize their own outputs and that self-recognition correlates with preferring them as evaluators, so a candidate whose family wrote or grades the test can score higher for reasons unrelated to quality.

Why does contamination matter more for evaluation than for training?

In training, overlap with the test inflates a score you can still check against fresh data; in evaluation, overlap compromises the measurement itself and there is no fresher check behind it. Contamination can also happen during the test, when an agent searches for answers that have been published.

How can an investor check a company's synthetic evaluation claims?

Ask for the three properties separately: evidence that items resemble deployment cases, the model families behind item generation and grading, and how held-out cases stay sealed. The strongest evidence compares rankings from the synthetic set with rankings from a sealed real set, using real split-half agreement as the benchmark.

Does SourceX publish evaluation data or deal metrics?

No. SourceX's deal, revenue and commercial figures are not public, and the company shares company and commercial metrics with investors directly. Investor materials can be requested at sourcex.si/investors, and any offering is made only to verified accredited investors.

Sources

  • NVIDIA's Nemotron-4 340B technical report (June 2024) says over 98 percent of the data used in its model alignment process was synthetically generated. Source
  • Panickssery, Bowman and Feng (NeurIPS 2024) report that LLMs such as GPT-4 and Llama 2 distinguish their own outputs with non-trivial accuracy and that fine-tuning reveals a linear correlation between self-recognition and self-preference. Source
  • Chen et al. (2025) report that stronger models exhibit greater self-preference and that much of this preference aligns with objectively superior performance. Source
  • The Humanity's Last Exam authors publicly released a large question set while maintaining a private test set to assess potential model overfitting. Source
  • A 2025 study of search-time data contamination found that for approximately 3 percent of Humanity's Last Exam questions, search-based agents directly find datasets with ground-truth labels on Hugging Face. Source
  • LiveBench's README says it limits potential contamination by releasing new questions monthly, scores questions against verifiable objective ground truth without an LLM judge, and does not make all questions of the current release public. Source
  • Article 10(3) of Regulation (EU) 2024/1689 sets quality criteria for the training, validation and testing data sets of high-risk AI systems, including relevance, sufficient representativeness, and freedom from errors and completeness as far as possible, judged against the intended purpose. Source
  • Rahmani et al. (SIGIR 2024) report that LLM-built synthetic test collections, with generated queries and judgments, can reliably be used for retrieval evaluation, and examine the risk of bias toward LLM-based systems. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies