AI data market
Synthetic data negative control tasks: a protocol for testing claimed gains
By SourceX Editorial · Updated
10 sections
Short answer
Test a claimed synthetic data improvement by training control arms beside the treatment: compute-matched real replay, shuffled-label synthetic, format-only synthetic and generator-blind synthetic, then score every arm on an untouched real holdout and on negative control tasks the synthetic data should not affect. Trust the gain only if it is specific to the target task. What follows is a method for running that test, not a report of any outcome.
Evaluation split
Synthetic data negative
Key takeaways
- This is a protocol page: it describes how to test a synthetic data gain and reports no results.
- Extra compute, format imitation, label shortcuts, generator leakage and seed variance can each fake an improvement.
- Four control arms (compute-matched replay, shuffled labels, format-only, generator-blind) give each rival explanation its own test.
- Negative control tasks share the target's format and grader but should not improve; if they do, the gain is not specific.
- Every arm is scored per item on real cases the generator never saw, with intervals across several seeds.
- Passing all controls narrows the explanations; it does not prove the absence of bias.
What does a negative control reveal about a synthetic data gain?
A negative control is an arm or a task that shares every plausible source of bias with your main comparison but cannot benefit from the thing you are testing. If the control moves as much as the treatment, the gain came from somewhere other than the synthetic content. This page sets out a protocol for running those controls; it reports no results and implies none.
The idea comes from epidemiology. There, a negative control outcome is one influenced by the same unobserved confounders as the outcome of interest but not directly caused by the exposure, so an association with it signals bias. Later work extended negative controls to selection and measurement bias in both observational studies and randomized trials, which matters here because most synthetic data comparisons are small randomized experiments with many hidden choices.
Translated to model training, the exposure is the synthetic mixture, the outcome is held-out task performance, and the confounders are everything that changed alongside the synthetic rows: more optimizer steps, a different prompt format, a generator that saw benchmark text, a lucky seed, or labels that carry a shortcut. A negative control isolates one of those at a time.
Which explanations other than the synthetic content can produce the gain?
At least six alternative explanations can produce a better score after adding synthetic data, and each one needs its own control. The table pairs each with the arm or check that exposes it, so a team can see which controls its current comparison already lacks.
| Alternative explanation | How it fakes an improvement | Control that exposes it |
|---|---|---|
| Extra training compute | The treatment run takes more steps or tokens than the baseline, so it is simply trained longer | Compute-matched real replay arm with identical steps and tokens |
| Format or style imitation | Synthetic rows teach the output template the grader rewards, not the task skill | Format-only arm: same templates and lengths, content from an unrelated domain |
| Generator leakage of test items | The generator reproduces or paraphrases evaluation items it saw during its own training | Generator-blind arm plus overlap screening of synthetic rows against the holdout |
| Shortcut cues in labels | Generated answers share surface cues with the correct label, and the model learns the cue | Shuffled-label arm and a partial-input baseline on the holdout |
| Seed and run variance | One favorable run is reported; the difference sits inside normal run-to-run spread | Several seeds per arm, paired per-item comparison with intervals |
| Evaluation drift | The holdout, grader or prompt changed between the baseline and treatment runs | Frozen, versioned holdout and grader hashed before the first run |
Which six arms does the negative control design train?
Run six arms from the same base checkpoint, the same real training set, the same hyperparameter search budget and the same evaluation harness. Only the synthetic component differs. The treatment arm is the one you want to believe; the four control arms are there to give every rival explanation its own chance to produce the same gain.
The shuffled-label arm borrows from a well-known result: networks trained on randomly relabeled data can reach zero training error while test error stays at chance. If shuffled synthetic labels still lift the holdout score, the lift is coming from inputs, format or extra steps, not from correct supervision. The format-only arm answers the same question for templates and length.
| Arm | Training data | Held fixed against treatment | A gain in this arm means |
|---|---|---|---|
| A. Real baseline | Real training set only | Base checkpoint, harness, search budget | Reference point for every comparison |
| B. Treatment | Real set plus the synthetic mixture at the proposed ratio | Everything except the synthetic rows | The claim under test; trusted only if controls C to F fall short of it |
| C. Compute-matched replay | Real set replayed or extended with more real rows to match B's steps and tokens | Steps, tokens, schedule | Part or all of B's gain is extra training, not synthetic content |
| D. Shuffled-label synthetic | Same synthetic inputs as B with outputs permuted across rows | Inputs, format, volume, steps | The gain comes from inputs or exposure, not correct answers |
| E. Format-only synthetic | Same templates, lengths and volume as B, content from an unrelated domain | Format, length, volume, steps | The grader rewards form; the task skill may not have improved |
| F. Generator-blind synthetic | Synthetic rows from a generator and grounding set with no access to holdout sources, screened for overlap | Recipe, ratio, volume | If B beats F by a wide margin, suspect leakage into B |
Which negative control tasks belong in the evaluation suite?
Add two or three tasks the synthetic mixture has no plausible reason to improve, scored in the same harness and format as the target task. If the treatment lifts those tasks as much as the target, the improvement is not specific, and the likeliest causes are format learning, extra compute or a grader artifact.
At a glance
Which negative control tasks belong in the evaluation suite
Pick negative control tasks deliberately. They should share the prompt shape, answer length and grader with the target, so they pick up the same artifacts, but require knowledge or decisions the synthetic rows never contain. In a claims-routing study, for example, an illustrative negative control task could be routing for a product line absent from the generator's grounding records.
Also add a partial-input check on the target holdout. In natural language inference, a classifier that saw only the hypothesis, not the premise, beat the majority baseline, exposing annotation artifacts. Generated data can carry the same kind of cue, so score a model that sees only part of each holdout input; if it does well, the holdout rewards shortcuts and the synthetic gain may be one.
- Target task: the real business task the synthetic data was built for, on the untouched holdout.
- Negative control task 1: same format and grader, different domain the generator was not grounded in.
- Negative control task 2: a capability the base model already has, to catch broad regressions or broad inflation.
- Partial-input baseline: the target holdout with a key input removed, to measure shortcut availability.
- Coverage audit: holdout cases grouped by exception type, to show which real conditions the synthetic mixture never represented.
How should the population be sampled and the real holdout kept untouched?
The holdout must be real cases the generator, its grounding records and its prompt authors never touched, drawn by a documented rule from a stated population and period. Without that, every arm is scored against data that may already resemble the synthetic rows, and no control can rescue the comparison.
State the population as precisely as a reader would need to rerun it: which systems of record, which business units, which date range, which case types. Sample by stratum so rare but important exceptions appear in usable numbers, and record the denominator for every stratum. A holdout that only contains routine cases will hide exactly the failures synthetic data tends to miss.
Disclose the derivation of the synthetic side with the same care: generator and version, grounding sources and their dates, the generation prompt or recipe, filters, how rows were selected, and the mixture ratio. Screen synthetic rows against the holdout for exact and near duplicates. Deduplication research found train-test overlap affecting over 4% of the validation sets of standard datasets, and a separate statistical test can flag benchmark contamination in black-box models without access to their training data, so treat overlap as something to measure, not assume away.
Then answer the coverage question directly. Group holdout cases by real exception or condition, such as a reversed payment, a partial shipment or a policy change mid-case, and record which groups have no synthetic counterpart. Report treatment and control scores per group; a mixture can raise the average while doing nothing for the conditions that matter most.
Which variables, metrics and decision rule settle the claim?
Fix the primary metric, the comparison and the decision rule before training any arm. The primary comparison is B minus A on the target holdout; the claim survives only if B also beats C, D, E and F by a margin whose interval excludes zero, and if B's gain on the negative control tasks is clearly smaller than its gain on the target.
| Variable | Definition | How it is reported |
|---|---|---|
| Target gain | Per-item score difference B minus A on the untouched holdout | Mean paired difference with an interval |
| Control gaps | B minus C, B minus D, B minus E, B minus F on the same items | One paired difference and interval per control |
| Specificity | B's gain on the target minus B's gain on each negative control task | Difference with an interval; near zero means non-specific |
| Shortcut availability | Partial-input model score on the target holdout versus majority baseline | Both scores, same items |
| Coverage by exception group | Target gain within each exception or condition stratum | Per-stratum difference with denominator |
| Overlap rate | Share of synthetic rows matching a holdout item exactly or near-exactly | Count, denominator and matching rule |
How much uncertainty is too much to call it an improvement?
If the interval on the paired difference between treatment and its strongest control includes zero, report no demonstrated improvement, whatever the point estimate says. Run several seeds per arm, compare per item rather than per aggregate score, and correct for the number of control comparisons.
Run-to-run variance is not a footnote. A study of machine learning benchmarks found that variance from data sampling, parameter initialization and hyperparameter choice noticeably affects results, and recommended accounting for those sources in comparisons. Seeds and the hyperparameter search budget should therefore be identical across arms, and seeds should be listed.
For the interval itself, paired bootstrap resampling of holdout items is a common, well-documented choice; it was set out for comparing machine translation systems and generalizes to any per-item metric. Whatever test you choose, name it, the resampling unit, the number of resamples and the correction for multiple controls in the pre-registration.
Decide the minimum effect worth acting on in advance, in the units of the business task, such as correctly routed cases per hundred. A gain smaller than that threshold is not a reason to buy or generate more synthetic data even if it clears statistical significance.
What must be pre-registered before the first training run?
Write and date the plan before any arm trains, and keep exploratory analyses labeled as such. Preregistration is meant to separate confirmatory analyses, which test planned hypotheses, from exploratory ones that generate new hypotheses; it does not forbid exploration.
At a glance
What must be pre-registered before the first training run
- Research question stated as a single comparison: does this synthetic mixture improve this task on real cases beyond what compute, format, labels and leakage explain?
- Population, period and sampling rule for the real holdout, with stratum definitions and target counts.
- Holdout and grader version hashes, frozen before generation begins.
- Generator, version, grounding sources with dates, generation recipe, filters and mixture ratio.
- The six arms, their data, and what each holds fixed; base checkpoint and hyperparameter search budget per arm.
- Negative control tasks and the partial-input baseline, with reasons each should be unaffected.
- Overlap screening method and threshold, and what happens to matching rows.
- Primary metric, decision rule, minimum effect worth acting on, and the specificity criterion.
- Seeds per arm, interval method, resampling unit and multiple-comparison correction.
- Privacy screen for real records and for synthetic rows that may reproduce them.
- Reporting commitment: all arms and all negative control tasks reported, including those that undercut the claim.
What can a clean set of controls still not prove?
Passing every control shows the gain is not explained by the rivals you tested; it does not prove the gain is real in production or that no other bias exists. Epidemiologists note that a null negative control result can never fully prove the absence of bias, and the same limit applies here.
The protocol tests one generator, one mixture ratio and one base model at one point in time. It says nothing about repeated training on model-generated data across generations; research published in Nature describes a degenerative process in which models trained recursively on generated data lose the tails of the original distribution first. If your pipeline will feed model outputs back into training, that needs its own longitudinal design.
Finally, the holdout defines what counts as improvement. If it under-represents a customer segment, a language or a rare exception, the protocol will certify a gain that does not exist for them. Treat the coverage audit as part of the result, not an appendix.
Where do the real cases for holdouts and controls come from?
Every arm in this design depends on real cases that never touched the generator: the holdout, the stratified exception groups and the grounding records for negative control tasks. Teams that lack them internally often license them from operating businesses.
With SourceX, the untouched holdout can be written into the request itself: its request-data form takes the scope, volume, language, format, timing and licensing needs, so a team can name the system, period and case types, and SourceX then finds matching records at U.S. businesses rather than handing over a catalog set the generator might already have seen. The supplying business approves the scope and terms and signs before delivery, which documents the holdout's provenance from day one.
Frequently asked questions
Does this page report whether synthetic data improves models?
No. It publishes the method only: the arms, negative control tasks, holdout rules, metrics and uncertainty rules. Any result would need collected evidence under a dated pre-registration, with every arm reported.
What is the single most important control if I can only afford one?
The compute-matched real replay arm, because extra training steps are the most common rival explanation and the cheapest to rule out. It still leaves leakage and format learning untested, so state that limit in the report.
How is a negative control task different from a regular benchmark?
A regular benchmark measures a capability you hope improves. A negative control task is chosen because the synthetic data should not improve it, while sharing the format and grader of the target task. Movement on it is a warning sign.
Which real exceptions does a synthetic mixture usually miss?
That has to be measured per study, not assumed. The protocol groups holdout cases by exception type and reports which groups have no synthetic counterpart and how the treatment performs on each.
Can the generator's own outputs serve as the holdout?
No. A holdout drawn from the generator shares its blind spots and style, so it rewards the treatment by construction. The holdout must be real cases the generator, its grounding records and its prompt authors never saw.
Sources
- A negative control outcome is influenced by the same unobserved confounders as the outcome of interest but is not directly influenced by the exposure. Source
- Negative controls can be structured to detect selection bias and measurement bias in both observational studies and randomized trials. Source
- A classifier that sees only the hypothesis can beat a majority baseline on natural language inference data, revealing annotation artifacts. Source
- Neural networks trained on randomly labeled data reach zero training error while test error stays at chance. Source
- Training recursively on model-generated data causes model collapse, in which the tails of the original distribution disappear first. Source
- A statistical test can detect test set contamination in black-box language models without access to training data or weights. Source
- Train-test overlap affects over 4% of the validation sets of standard datasets, and deduplication reduces it. Source
- Preregistration distinguishes confirmatory from exploratory analyses and does not forbid unplanned analyses. Source
- Variance from data sampling, parameter initialization and hyperparameter choice markedly affects machine learning benchmark results. Source
- Paired bootstrap resampling can estimate significance when comparing two systems on the same test items. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.