AI uses for records
How to measure synthetic data marginal utility before generating more
By SourceX Editorial · Updated
10 sections
Short answer
To measure diminishing returns from additional synthetic examples, hold the real training set fixed, add generated examples in doubling increments, train repeated seeds at each step and score every model on an untouched real holdout. Estimate the paired gain from each increment with intervals, compare it with a repeat-only control and a recipe change, and stop when gains fall below a pre-declared threshold. This page is a protocol; it reports no results.
Value drivers
Synthetic data marginal
Key takeaways
- Marginal utility is the change in real-holdout performance from the next increment of synthetic examples, with the real training set held fixed.
- A doubling ladder with repeated seeds and paired differences between adjacent steps is the core design; one large run per size cannot show diminishing returns.
- A repeat-only control separates the value of new generated content from the value of more optimization steps.
- Read the curve by slice: rare real exceptions usually stop improving before the average does.
- Pre-declare the minimum useful gain and the stopping rule before any run, and report no curve without its intervals.
How do you measure diminishing returns from additional synthetic examples?
You measure it as a slope, not a level. Keep the real training records fixed, add synthetic examples in steps that double in size, train several seeds at each step and score every model on real tasks the generator never saw. The quantity you want is the gain from one step to the next, with an interval around it. When that gain is reliably smaller than the smallest improvement you would act on, more of the same generation is not worth it.
This page is a protocol for running that measurement. It reports no results, curves or thresholds from any experiment; every number in it is a design parameter you set before running. It differs from a mixture experiment that varies real and synthetic counts together: here the real set is held constant and the question is only whether the next batch of generated data earns its compute, review and disclosure costs.
Why the next increment can add little
There are three separate reasons to expect flattening, and the experiment should be able to tell them apart. The first is ordinary scaling. Hestness and colleagues found that generalization error falls as a power law in training set size across several domains, and that better architectures shift the curve without changing its exponent, so each doubling buys less than the last.
The second is that learning curves do not always behave. Viering and Loog's review finds that well-behaved curves often look like power laws or exponentials but that no single model fits all of them, and their later survey documents cases where more data leads to worse performance. A synthetic increment can sit on such a stretch.
The third is specific to generated data. Generated examples come from a narrower distribution than the records they imitate, and He and colleagues identify the domain gap between generated and real images as a main limit on the gains from synthetic training data. More volume from the same recipe repeats the same gap, so the curve can flatten well before a real-data curve would.
The marginal utility learning curve experiment
The design borrows the standard learning curve, a plot of validation score against the number of training samples used to see how much more data helps, and specializes it to one variable: the synthetic count at a fixed real set. Six arms make the result interpretable.
Pick the ladder so the smallest synthetic step is large enough to move the metric at all and the largest is the most you would realistically generate. Six to eight doubling steps above zero is one workable shape, but the exact sizes are yours to declare in advance.
| Arm | What changes | What stays fixed | What it isolates |
|---|---|---|---|
| A. Real-only baseline | Nothing; synthetic count is zero | Real training set, model, training budget rules | The starting point every gain is measured from |
| B. Synthetic doubling ladder | Synthetic count doubles at each step | Real set, generator version, recipe, filters | Marginal utility of more of the same generation |
| C. Repeat-only control | Training steps rise to match each ladder step, using the smallest synthetic set repeated | Real set and the content of the synthetic set | Whether gains come from new content or just more optimization |
| D. Recipe change at matched count | Generation recipe: new grounding records, prompts or diversity controls | Synthetic count at a chosen ladder step | Whether changing how data is made beats making more of it |
| E. Real increment comparison | Adds a small block of extra real records instead of synthetic | Everything else at the baseline | What a unit of real data is worth relative to a synthetic step |
| F. Shuffled-label negative control | Synthetic labels or targets are permuted at one ladder step | Count and inputs | That the pipeline and metric can detect harmful data |
Which real tasks, sampling and holdout keep the curve honest?
The population is the real work the model will face, and the holdout must come from it untouched. If any holdout record, or a near duplicate of one, was used to seed, prompt or filter the generator, the curve measures memorization, not utility.
- Split real records by time where possible: generator grounding and real training data from earlier periods, holdout from a later one, with the policy or rule date recorded for each.
- Stratify the holdout so rare but important exceptions have enough cases to score on their own, and declare the slice definitions before any run.
- Screen the synthetic set against the holdout for exact and near duplicates and log the screen's method and threshold.
- Screen generated examples for leaked personal data from grounding records before training, and record the screen result per batch.
- Freeze the holdout, the scoring code and the generator version for the whole ladder; a change mid-ladder restarts it.
Variables, metrics and what each run records
Each training run produces one row. The marginal gain is computed afterward from pairs of rows, never estimated inside a single run.
| Item | Definition | Recorded as |
|---|---|---|
| Synthetic count | Generated examples in the training set at this ladder step | Integer, plus synthetic share of all training examples |
| Generation record | Generator and version, grounding sources, prompt or recipe ID, filters applied | IDs and hashes per batch |
| Training budget | Steps or epochs and the rule linking them to data size | Declared rule, actual steps run |
| Primary metric | Task metric on the full real holdout | Value per seed |
| Slice metrics | Same metric on each pre-declared exception slice | Value per seed per slice |
| Marginal gain | Paired difference in the primary metric between a step and the step below, same seed | Mean and interval across seeds and holdout resamples |
| Cost of the step | Generation, review and training compute hours for the increment | Hours, not money, so results transfer between teams |
Estimating the marginal gain and its uncertainty
Diminishing returns are a claim about small differences, so variance decides whether the claim is real. Bouthillier and colleagues show that data sampling, parameter initialization and hyperparameter choice each move benchmark results markedly; a single run per ladder step can show a curve that bends only because of seed noise.
At a glance
Estimating the marginal gain and its uncertainty
Run at least several seeds per step, pairing seeds across steps so that step k and step k plus one share initialization and data order where the framework allows. Compute the gain per seed as a paired difference, then report its mean with an interval from a bootstrap over seeds and holdout examples together. Plot every step with its interval; do not fit and report a smooth curve without the points behind it.
Fitting a power law or a saturating curve to the ladder is optional and only descriptive. Because real learning curves can be ill-behaved, extrapolating a fitted curve beyond the largest step measured is a forecast, and should be labeled as one.
- Pre-declare the minimum useful gain: the smallest primary-metric improvement that would change a decision.
- Stopping rule: stop the ladder when the upper end of the interval on the marginal gain falls below the minimum useful gain at two consecutive steps.
- Novelty check: if the ladder's gain at a step is not distinguishable from the repeat-only control, new content is not what is helping.
- Recipe check: if arm D's gain over the matched ladder step exceeds the minimum useful gain, change the recipe before generating more.
Reading the curve by slice: where generated examples stop helping first
Averages hide the cases that matter most. A generator tends to reproduce the common patterns in its grounding records well and the rare ones poorly, and Shumailov and colleagues show that training indiscriminately on model-generated content can erase the tails of the original distribution. The practical consequence is that the curve for rare exception slices often flattens, or even turns down, while the overall curve still rises.
Plot each pre-declared slice separately with its interval. If an exception slice stops improving at a step where the average keeps climbing, more volume is buying gains on routine cases at the cost of the ones that justify the model. Those are the conditions to send back to the recipe arm, or to real records: the exception types the synthetic mixture does not represent, named and counted.
Running the ladder in practice: order of work and what to log
Run the arms in an order that lets you stop early without wasting compute. The baseline and the negative control come first, because if the pipeline cannot detect deliberately broken data, nothing later can be trusted. The ladder then runs from the smallest step upward, and the stopping rule is checked after each step rather than at the end.
Keep generation separate from training. Generate the largest planned synthetic pool once, assign every example a stable ID and draw each ladder step as a nested subset, so step k plus one contains all of step k. Nested subsets make the paired differences meaningful: the only change between adjacent steps is the added examples, not a fresh random draw.
- Freeze and hash the real training set, the holdout, the scoring code and the generator version.
- Train the real-only baseline with every planned seed and confirm the spread between seeds is small enough for the minimum useful gain to be detectable.
- Run the shuffled-label control at one mid-ladder step and confirm the holdout metric drops; if it does not, fix the pipeline before continuing.
- Generate the full synthetic pool once, run the duplicate, leakage and privacy screens, and log the counts removed by each screen.
- Train ladder steps from the smallest upward, pairing seeds, and compute the marginal gain and its interval after each step.
- Run the repeat-only control at the same training steps as each completed ladder step.
- Run the recipe-change arm at the ladder step where the gain first approaches the minimum useful gain.
- Apply the stopping rule, then run the real-increment comparison at the baseline to put the last useful synthetic step in context.
- Record every deviation from the pre-registered plan with its date and reason.
Pre-registration checklist for a marginal utility study
Write these down, date them and store them with the code before the first run. Anything changed afterward is reported as a deviation.
At a glance
Pre-registration checklist for a marginal utility study
Checklist · 9 items
- Research question: the task, the fixed real training set and its size, and the decision the result will inform.
- Ladder: every synthetic count to be tested, the generator version and recipe, and the training budget rule.
- Arms: which of the six arms will run, at which ladder steps the recipe and negative controls sit.
- Holdout: source, time window, size, slice definitions and the leakage and privacy screens applied.
- Metrics: primary metric, slice metrics, and how marginal gain is computed.
- Uncertainty: number of seeds, pairing rule, bootstrap design and interval level.
- Thresholds: minimum useful gain, stopping rule and the novelty and recipe checks.
- Disclosure: what will be published about the generation recipe, grounding sources and any data that cannot be shared.
- Statement that no result will be reported without its intervals, its negative control and its deviations.
0 of 9 ticked. Tick each item as you go.
What the protocol cannot settle, and where real records come in
The experiment answers a narrow question for one task, one generator and one fixed real set. It does not say the result carries over to another model size, another generator version or another domain, and a flat curve says nothing about whether a different recipe would help unless arm D was run. It also cannot measure what the holdout does not contain; if the real records lack an exception type, no curve will show it missing.
Arm E and the slice readings often point to the same next step: more real examples of the cases the generator cannot imitate. SourceX can look for a U.S. business that holds those exception records once a team asks, since it keeps no catalog; the request lists scope, volume, format, timing and licensing needs, and for a marginal utility study the scope is the exception types and the time window the holdout and the generator's grounding require. Records reach the team only after their owner has approved the scope and signed.
Frequently asked questions
What is the marginal utility of synthetic data?
It is the change in performance on real, untouched tasks that the next increment of synthetic training examples produces, with the real training data held fixed. It usually shrinks as the synthetic count grows, and it can turn negative for rare cases.
Why include a repeat-only control?
Adding synthetic examples also adds training steps. Repeating a small synthetic set for the same number of steps shows how much of the gain comes from optimization alone, so you can tell whether new generated content is what helps.
What baseline and sample size make a diminishing-returns claim defensible?
A real-only baseline, several paired seeds at every ladder step, a holdout large enough to score each declared slice and intervals on every marginal gain. Without a negative control and pre-declared thresholds, a bend in the curve can be noise.
Should we fit a power law and extrapolate?
Fitting is fine as a description of the measured steps. Extrapolating beyond the largest step is a forecast, since learning curves can be ill-behaved, and it should be labeled as one with its assumptions.
Does this protocol include measured marginal gains?
No. It describes the design, controls, metrics and stopping rule only; no curves or gains are reported. Any numbers in the design, such as the number of ladder steps, are parameters to set before running.
Sources
- Generalization error follows power-law learning curves in training set size across domains, and better architectures shift the curve without changing its exponent. Source
- Learning curves often resemble power laws or exponentials, but ill-behaved curves exist and no single model fits all of them. Source
- A survey of learning curves documents cases where more training data does not lead to better performance. Source
- Indiscriminate training on model-generated content causes defects in which the tails of the original distribution disappear. Source
- Synthetic data from generative models helps most when real data is scarce, with gains limited by the domain gap to real data. Source
- Variance from data sampling, parameter initialization and hyperparameter choice markedly affects machine learning benchmark results. Source
- A learning curve plots validation and training scores against the number of training samples to show how much more data helps. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.