Skip to content

AI data market

How to measure how much real data a synthetic dataset needs

By SourceX Editorial · Updated

10 sections

Short answer

How much real data a synthetic dataset needs has no general answer; it has to be measured for each task and generator. This protocol shows how: a synthetic-real data mixture experiment that trains matched models across a grid of real and synthetic counts and scores every one on untouched real tasks, with negative controls, repeated seeds and paired uncertainty. It reports no results.

Value drivers

Synthetic real data mixture

This page covers how to measure how much real data a synthetic dataset needs.Illustrative

Key takeaways

  • No general real-to-synthetic ratio is supported across tasks; measure it per task and generator.
  • Compare a real-only learning curve with a mixed curve at the same real counts, scored on a locked real holdout.
  • Matched-volume, ungrounded and corrupted-label controls show whether the synthetic data itself is doing the work.
  • Use several seeds and paired, clustered intervals on item-level differences.
  • Report rare-exception cohorts separately; that is where synthetic mixes most often fall short.

How much real data does a synthetic training dataset need?

There is no published constant for how much real data a synthetic dataset needs; the amount depends on the task, the generator, how the synthetic data was grounded and how far the target distribution sits from what the generator learned. The defensible way to answer it for your use case is to run a controlled synthetic-real data mixture experiment and read the result off a curve measured on real held-out tasks.

This page publishes the method only. It is a protocol, not a study: no results are reported, and any numbers below are design parameters to adjust, not findings. A team can run it on a classification, extraction or instruction-following task with its own generator and its own real records.

The research literature explains why the question matters. A Nature paper by Shumailov and colleagues reports that models trained on recursively generated data lose the tails of the original distribution first. Work by Gerstgrasser and colleagues reports that replacing real data with synthetic data tends toward collapse while accumulating real and synthetic data avoids it in their settings, and Alemohammad and colleagues report that self-consuming loops without enough fresh real data decline in quality or diversity. None of those papers tells you the right real share for a given business task, which is what the experiment measures.

What question does the mixture experiment answer?

The experiment answers two linked questions for one target task: how does held-out real performance change as you add real examples to a fixed synthetic set, and does the synthetic set add anything beyond the real examples alone. The second question is the one most often skipped, and it decides whether the synthetic data is worth its cost.

At a glance

What question does the mixture experiment answer

Write the question down in a form that the analysis can answer without interpretation. For example: for invoice line-item extraction on the team's held-out real invoices, what is the smallest real count at which adding the synthetic set no longer improves field-level accuracy over real-only training, at the pre-registered confidence level. The task, the metric, the population and the decision rule are all named before any model is trained.

  • Primary estimand: the difference in held-out real performance between real plus synthetic and real-only, at each real count.
  • Secondary estimand: performance of synthetic-only training on the same real holdout, the train-on-synthetic, test-on-real comparison introduced by Esteban, Hyland and Ratsch.
  • Cohort estimand: the same differences restricted to rare exceptions in the real holdout, where synthetic data is most likely to be thin.
  • Regression estimand: change on a fixed suite of capabilities the model already had, so a gain on the target task is not bought with a loss elsewhere.

Mixture experiment design: arms and what each one isolates

The core of the protocol is a grid of training arms that vary only the training data. Every arm starts from the same base model with the same hyperparameters and the same training budget, and every arm is scored on the same untouched real holdout.

SDMetrics, the open-source synthetic data metrics library, implements the simplest version of this comparison for tabular classifiers: it trains one model on real training data alone and one on real training data concatenated with synthetic data, scores both on real validation data, and reports the difference. The protocol below extends that pairwise check into a dose-response grid with controls.

Mixture experiment design: arms and what each one isolates
ArmReal examplesSynthetic examplesWhat it isolates
A. Real-only ladderEach step of the real grid (for example 0, 250, 1,000, 4,000; illustrative)0Learning curve from real data alone; the baseline every mixed arm is compared with
B. Mixed ladderEach step of the same real gridFixed full synthetic setAdded value of the synthetic set at each real count
C. Synthetic-only0Full synthetic set, plus half and quarter subsetsTrain on synthetic, test on real; how far synthetic data goes on its own
D. Matched-volume controlEach real step, with real examples repeated to equal arm B's example count0Whether arm B wins only because it trained on more examples or tokens
E. Ungrounded synthetic controlEach real stepSame count generated from generic prompts without your grounding recordsWhether grounding in your records is what makes the synthetic data useful
F. Corrupted-label controlEach real stepSame synthetic inputs with shuffled or randomized labelsA negative control; if this arm matches arm B, the evaluation is not sensitive to label quality
G. Ratio sweep at fixed totalReal share varied across the gridRemainder of a fixed total budgetThe best real share when total training volume is capped

Which population, sampling and holdout rules keep the result valid?

The holdout must be real, untouched and drawn from the population the model will serve. Untouched means the generator never saw it: not as seed examples, not as retrieval context and not as part of the grounding records used to write prompts or rules.

At a glance

Which population, sampling and holdout rules keep the result valid

Sample the holdout first and lock it before any synthetic data is generated. If the records carry dates, use a time split so the holdout comes from a period after every record the generator or the real training pool used; that protects against leakage through near-duplicate records and shows how the model handles drift. Stratify the holdout so rare exceptions appear in numbers large enough to report separately, and record the denominator for every cohort.

  • Population: the real records the deployed model will see, defined by system, period, geography and record type.
  • Real training pool: drawn from the same population before the holdout cutoff; each real-count step is a nested random subset so larger steps contain smaller ones.
  • Grounding records: the real records used to seed or condition the generator, listed by id and kept out of the holdout.
  • Holdout: locked, stratified by cohort, with a written cohort definition and denominator.
  • Deduplication: near-duplicate checks between the holdout and both the real pool and the synthetic set, with the method and threshold recorded.

Variables, metrics and the disclosure record

Fix everything except the training data, and write down every variable before training starts. A mixture result is only interpretable if a reader can see the generator recipe, the grounding and the policy date that the synthetic data reflects.

Synthetic business data inherits the rules that were true when it was generated. If refund policy, tax rules or approval limits changed after the generator's grounding records were collected, the synthetic set teaches the old rule, so the disclosure records a policy date and a rule-consistency check.

Variables, metrics and the disclosure record
VariableRoleHow it is fixed or recordedWhy it matters
Real count and synthetic countIndependentSet by the arm gridThe quantities the experiment varies
Base model, hyperparameters, training budgetControlledIdentical across arms; budget fixed in tokens or stepsOtherwise differences reflect training, not data
Random seedControlled and repeatedSeveral seeds per arm, same seed list for every armSeed alone can move fine-tuning results
Generator recipeRecordedModel, prompt templates, sampling settings, filters, versionResults apply only to this recipe
Grounding sources and policy dateRecordedRecord ids, date range and the business rules in forceSynthetic data can encode outdated rules
Primary metric on real holdoutOutcomePre-registered metric and scoring scriptThe decision is made on this number
Cohort and regression metricsOutcomeSame scoring on rare-exception cohorts and the regression suiteShows where synthetic data fails and what it breaks

How should uncertainty be reported?

Report differences between arms with intervals, computed on paired item-level scores, and repeat every arm across several seeds. Dodge and colleagues found that changing only the random seed in fine-tuning produced substantial variation in results, so a single run per arm cannot separate a data effect from seed noise.

At a glance

How should uncertainty be reported

Miller's paper on adding error bars to evaluations recommends computing standard errors, using clustered standard errors when questions come in related groups, and comparing two models through inference on question-level paired differences rather than on the two summary scores. In this protocol every arm is scored on the same holdout items, so the comparison between arm B and arm A at each real count is a paired difference per item, clustered by source record or customer when several items come from one.

Present the result as two curves over the real-count grid, real-only and mixed, with their difference and its interval at each step. The pre-registered decision rule then reads directly off the difference curve.

  • Seeds: the same seed list for every arm; report the spread across seeds alongside the interval.
  • Pairing: item-level differences between arms on the identical holdout.
  • Clustering: by the unit that makes items correlated, such as a customer, a document or a ticket thread.
  • Multiple comparisons: name in advance which comparisons are confirmatory; label the rest exploratory.
  • Cohorts: report every pre-registered cohort, including those where the synthetic set hurts.

Which real exceptions does the synthetic mixture fail to represent?

Answer this with the cohort analysis, not by inspecting the synthetic data. If the mixed arm improves the overall metric but not the rare-exception cohorts, the synthetic set is filling in the common cases the real data already covered.

Look first at the cases the collapse literature predicts will be lost: low-frequency categories, long or unusual records, combinations of conditions and anything whose handling depends on a rule that changed. For a support workflow that might be partial refunds after a policy change; for accounts payable, a three-way match failure with a split shipment. List these cohorts in the pre-registration so they cannot be dropped later because they look bad.

When a cohort fails, the remedy is usually more real examples of that cohort rather than more synthetic volume, and the experiment can be rerun with a targeted real addition to confirm it.

How do you run the experiment step by step?

Run the protocol in a fixed order so that nothing learned from early results can leak into the design. The order below locks the holdout and the plan before the first synthetic example exists, and it trains the cheap control arms alongside the main arms rather than afterward, so they cannot be dropped if they turn out to be inconvenient.

Steps

How do you run the experiment step by step

Budget for the full grid before starting. Seven arms across several real counts and several seeds adds up quickly, and a common failure is to run the main arms, run out of compute and skip the controls that would have changed the interpretation. If compute is tight, reduce the number of real-count steps rather than the controls or the seeds.

Checklist · 10 items

  1. Define the task, metric, population and decision rule, and register the plan.
  2. Sample and lock the real holdout; record cohort denominators.
  3. Draw the real training pool and the nested real-count subsets from records dated before the holdout cutoff.
  4. Select grounding records from the training pool only and list their ids.
  5. Generate the synthetic set with the recorded recipe; run the privacy screen and the rule-consistency check; deduplicate against the holdout.
  6. Build the control sets: repeated real examples, ungrounded synthetic and corrupted labels.
  7. Train every arm with the same base model, hyperparameters, budget and seed list.
  8. Score every checkpoint with the registered scoring script on the holdout and the regression suite.
  9. Compute paired, clustered differences per real count and per cohort, then apply the decision rule as written.
  10. Publish or file the result with the disclosure record, the deviations from the plan and the stated limits.

0 of 10 ticked. Tick each item as you go.

Pre-registration checklist for a synthetic-real mixture experiment

Register the plan before generating synthetic data or training any arm. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan to a repository before data collection or analysis begins; an internal team can use the same discipline with a dated, access-controlled document.

Checklist · 10 items

  1. Research question, estimands and the decision rule, written as they will be evaluated.
  2. Target task, primary metric and the exact scoring script version.
  3. Population definition, holdout sampling method, cohort definitions and expected denominators.
  4. Real-count grid, synthetic subsets and the full arm list including controls D, E and F.
  5. Generator recipe, grounding record list, policy date and the rule-consistency check.
  6. Privacy screen applied to synthetic outputs, such as checks for copied personal details from grounding records.
  7. Base model, hyperparameters, training budget and the seed list.
  8. Uncertainty method: pairing, clustering unit, interval type and confirmatory comparisons.
  9. Stopping and exclusion rules: what counts as a failed run and how it is handled.
  10. Limits stated in advance: the result applies to this task, generator and data period only.

0 of 10 ticked. Tick each item as you go.

What are the limits, and where does licensed real data fit?

The protocol measures one task with one generator over one data period. A result does not transfer to a different generator, a different grounding set or a later policy period without a rerun, and a curve measured on a small holdout supports only coarse conclusions. These limits belong in any write-up of the result.

The real side of the experiment usually needs more real records than teams expect: a real training pool large enough for the top of the grid, grounding records, and a locked holdout the generator never touches. Each use needs rights that cover it, and synthetic data generated from licensed records is a derived dataset whose permitted uses depend on the license.

Teams short of that real side can ask SourceX to find it. Because SourceX holds no catalog, each real pool is located at U.S. businesses for that particular request, and the request-data form is where to state scope, volume, language, format, timing and licensing needs, including whether synthetic derivatives may be generated from the records. The contributing company approves scope and terms and signs before its records are handed over.

How do you measure real data marginal contribution when the generator and compute change too?

Measure it as a decomposition, not a before-and-after. When a pipeline moves to a new version, the real seed records rarely change alone: the generator is upgraded, more examples are drawn, training runs longer and filters are retuned. The protocol above varies the training mix under one generator; a claim that real data lifted a model across versions needs each of those changes isolated first.

Generator changes alone move results. Peng and colleagues kept the 52,000 Alpaca instructions and had GPT-4 write a new answer to each; in their early experiments the GPT-4 data gave instruction-tuned LLaMA models better zero-shot performance on new tasks than data generated by earlier models. The instructions, the seed side, never changed.

Compute is the second impostor. Muennighoff and Rush found that with data constrained and compute fixed, up to four epochs of repeated data changed loss negligibly compared with unique data, and that with more repetition the value of added compute decays toward zero. So every cell gets one token budget, with any repetition reported in epochs.

Seedless generation sets the bar. Magpie prompted aligned Llama-3 models with nothing but the chat template up to the user turn, generated 4 million instructions with responses and selected 300,000; the authors report that fine-tuning on that data alone beat earlier public datasets. Real seeds earn credit only for what they add over the best seedless pipeline, not over no synthetic data at all.

The ledger lists the changes that most often pass for real-data value between versions:

How do you measure real data marginal contribution when the generator and compute change too?
Change between versionsHow it can pass for real-data valueHold it fixed by
Generator model or versionA stronger generator writes better outputs from the same seedsPinning the generator, or entering it as its own factor
Generated volumeMore examples are drawn from the same seedsEqual generated counts in every cell
Training tokens or epochsLonger training lifts every armOne token budget per cell, with repetition reported
Filters and judgesA retuned filter removes errors the seeds get credit for preventingVersioned filters applied identically in every cell
Seed curation effortHand-picked seeds beat a random draw from the same sourceDrawing seeds by a written rule, with any curated draw as a separate cell
Evaluation setA new or leaked holdout shifts every scoreOne locked holdout across all versions

How should the gain be split when seeds, generator and budget interact?

Split it by averaging each change's contribution over every order in which the changes could have been made. Credit assigned one change at a time depends on the order, because a better generator can make better use of the same seeds. Ghorbani and Zou's Data Shapley uses the same logic for single training points, sampling random orderings and averaging each point's marginal contribution; here the players are a handful of pipeline factors.

Two factors, real seeds and the generator, need four cells. Adding the training budget as a third factor makes eight cells and six orderings, still few enough to run every cell rather than sample them.

Illustrative arithmetic, with invented scores that are not results: version one scores 60 on the locked holdout and version two scores 74. Adding licensed seeds first credits them with 4 points and the new generator with 10; upgrading the generator first credits it with 6 and the seeds with 8. Averaged over both orders, the seeds earn 6 points and the generator 8, and the 4-point interaction is split between them. A team quoting one order could claim 4 or 8 points for the same records.

How should the gain be split when seeds, generator and budget interact?
CellSeed inputGeneratorIllustrative score
Version onePublic proxy seedsOld generator60 (invented)
Seeds changed onlyLicensed real seedsOld generator64 (invented)
Generator changed onlyPublic proxy seedsNew generator66 (invented)
Version twoLicensed real seedsNew generator74 (invented)

What should a dated results report contain before anyone relies on the attribution?

It should contain every cell rather than the best comparison, with enough dates, versions and budgets for an outsider to rerun it. DCLM shows the discipline at benchmark scale: proposed training sets are judged by training a fixed recipe on each, at model sizes from 412 million to 7 billion parameters, across 53 downstream evaluations. A factor decomposition needs that same fixed recipe in every cell, run at the smallest scale that moves the metric, with the main contrast repeated at one larger scale. At minimum, the report lists:

  • Run dates and the version of every factor in every cell: seed set, generator, filters, training recipe and holdout.
  • The token budget per cell, with any repetition stated in epochs.
  • The full cell table, including cells that scored badly; one missing cell makes the averages unrecoverable.
  • Each factor's averaged share and the interaction, with intervals across repeated training seeds.
  • The seedless or public-seed pipeline used as the base, and why it is the strongest one available.
  • Rare-case cohorts scored separately, as already registered under the protocol above.
  • Dated deviations from the plan, and the scale at which each result was measured.

What would make a real-seed attribution misleading, and what should investors ask?

A seed share measured with today's generator is a dated reading, not a property of the records, because the interaction term is what moves when generators improve. Shares also depend on the holdout: a test set heavy in routine cases understates seeds whose value sits in exceptions. And seedless results like Magpie's mean the honest share for real seeds may be close to zero on some tasks.

Ask any company claiming its records lift models, SourceX included, for the full cell table behind the claim, the base pipeline it measured against, the generator version and token budget in each cell, and whether the share was rechecked after the latest generator change.

SourceX licenses business records that companies select and approve for AI use, so an attribution like this one bears on what such records are worth. SourceX has not run or published this decomposition, and its company figures go to investors only on request, through the investor page.

Methods note: the confound ledger, the factor decomposition and the report list were written on October 27, 2026 from the five publications cited, each checked through search excerpts on arXiv or the PMLR proceedings site. Scores in the worked example are invented, and no experiment was run for this page.

This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.

Sources for this section

  • Peng and colleagues regenerated answers to the 52,000 Alpaca instructions with GPT-4 and found the GPT-4 data gave instruction-tuned LLaMA models better zero-shot performance on new tasks than data from earlier models. Source
  • Muennighoff and Rush found that, with data constrained and compute fixed, up to four epochs of repeated data changed loss negligibly compared with unique data. Source
  • Magpie generated 4 million instructions from aligned Llama-3 models prompted with only the chat template, selected 300,000, and reports that fine-tuning on them beat earlier public datasets. Source
  • Ghorbani and Zou's Data Shapley estimates each training point's value by sampling random orderings of the data and averaging its marginal contribution. Source
  • DCLM judges proposed training sets by training a fixed recipe on each, at model sizes from 412 million to 7 billion parameters, across 53 downstream evaluations. Source

Frequently asked questions

Is there a standard ratio of real to synthetic data?

No general ratio is supported by evidence across tasks. The right amount depends on the task, the generator, the grounding and the distance to the target distribution, which is why this protocol measures it on a real holdout for each use case.

What should the controlled mixture experiment with held-out real tasks include?

It needs a real-only learning curve, a mixed curve at the same real counts, a synthetic-only arm, a matched-volume control, an ungrounded synthetic control and a corrupted-label negative control. Every arm uses the same base model, budget and seeds and is scored on the same locked real holdout.

Why not evaluate synthetic data with fidelity or similarity scores alone?

Similarity scores describe how closely synthetic data resembles real data, not whether it improves a model on real tasks. The mixture experiment measures the outcome that matters, held-out real performance, and the cohort analysis shows where the synthetic data falls short.

How big should the real holdout be, and what is the baseline?

Large enough that every pre-registered cohort, including the rare exceptions, has a usable denominator; there is no fixed size. The baseline is the real-only arm at each real count, scored with several seeds per arm and paired, clustered intervals on item-level differences, and the decision rule is registered before training.

Can the same real records seed the generator and test it?

No. Records used to ground or seed the generator must be excluded from the holdout, and near-duplicates should be removed, or the test rewards the generator for reproducing what it has already seen.

Does this page report results?

No. It publishes the design of the experiment only. Any figures shown are illustrative design parameters that a team should adjust to its own task and data.

Sources

  • A Nature paper by Shumailov and colleagues reports that training on recursively generated data causes models to forget the true underlying distribution, with the tails disappearing first. Source
  • Gerstgrasser and colleagues report that replacing real data with each generation's synthetic data tends toward model collapse, while accumulating real and synthetic data avoids it in their experiments. Source
  • Alemohammad and colleagues report that without enough fresh real data in each generation of a self-consuming loop, generative models decline in quality or diversity. Source
  • Esteban, Hyland and Ratsch introduced evaluating synthetic data by training a model on synthetic data and testing it on real held-out data (train on synthetic, test on real). Source
  • SDMetrics' data augmentation metrics train one classifier on real training data and one on real plus synthetic data, score both on real validation data and report the difference. Source
  • Miller recommends standard errors for evaluation scores, clustered standard errors when questions come in related groups, and inference on question-level paired differences when comparing two models. Source
  • Dodge and colleagues found that changing only the random seed when fine-tuning pretrained language models produced substantial variation in results. Source
  • The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan to a repository before data collection or analysis. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies