AI uses for records
Synthetic data validation holdout: keeping real test work untouched
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Validate synthetic training data on untouched real work by sealing a sample of real cases before any generation starts, then training on the synthetic set and scoring on those real cases. A written holdout policy decides who may see the cases, what counts as touching them, which comparisons they score and when a used holdout must be retired, so the score measures transfer to real work rather than echo.
Evaluation split
Synthetic data validation
Key takeaways
- Seal the real holdout before any generation prompt, grounding index or seed set exists, and record a hash and timestamp.
- Few-shot prompts, grounding indexes, filter tuning and guideline writing are all touches; reading one aggregate score is not.
- Score transfer, augmentation, copying and negative controls on the sealed cases, per exception type rather than only overall.
- Give the holdout a decision budget and rotate to a fresh sealed sample when it is spent or exposed.
- Every reported result should name the policy version, holdout period, strata, touch count and generation recipe.
How do you validate synthetic training data on untouched real work?
Seal a sample of real cases before anyone writes a generation prompt, keep it out of every step that shapes the synthetic set, and score a model trained on synthetic data against those sealed cases. The gap between that score and a model trained on comparable real data is the honest measure of what the synthetic set teaches.
The idea is not new. A 2017 paper on generating medical time series with recurrent GANs described evaluating generated medical time series by training on a synthetic labelled set and testing on real data, the setup now called train on synthetic, test on real. What has changed is how many ways the real test cases can leak into the generator: seed examples, retrieval grounding, filter tuning, prompt writers who read tickets for inspiration.
That is why the asset on this page is a policy, not a metric. A metric tells you the score. The policy tells a reviewer whether the score can be believed, because it records who could see the holdout, what they did with it and when it stopped being untouched.
What counts as touching the holdout, and what does not?
Any activity that lets information from a holdout case influence the generator, the grounding pool, the filters or the training choices counts as a touch. Reading an aggregate score does not, as long as the score is not used to tune the generator over and over.
Most leaks are quiet. A prompt engineer pastes three real tickets into a few-shot prompt to get the tone right. A retrieval index used for grounding is built from the whole case archive, holdout included. A quality filter threshold is picked because it maximizes the holdout score. Each makes the synthetic set look better on exactly the cases meant to judge it.
| Activity | Touch? | Why | Policy response |
|---|---|---|---|
| Using holdout cases as few-shot examples in a generation prompt | Yes | The generator sees the exact phrasing and outcomes it will be scored on | Retire the exposed cases and log the event |
| Building a grounding or retrieval index from an archive that includes holdout cases | Yes | Generated examples can paraphrase holdout content through retrieval | Rebuild the index from the development pool only |
| Choosing a quality filter threshold by its holdout score | Yes | The filter is selected to fit the holdout, which inflates it | Tune on a separate real development split |
| Writing annotation guidelines from holdout examples | Yes | Labels in the synthetic set inherit holdout specifics | Draw guideline examples from the development pool |
| A prompt writer reading holdout tickets for background | Yes | Memory of specific cases shapes scenarios | Restrict item access to the custodian and scorers |
| Scoring a finished model once and reading the aggregate result | No | Information flows out as a number, not into the generator | Count the run against the holdout's decision budget |
| Reading per-stratum results to plan the next generation round | Partial | Repeated rounds slowly fit the generator to the holdout | Allow a set number of rounds, then rotate |
The protected real-world holdout policy, clause by clause
The table below is the policy itself. Each clause says what is required, what evidence proves it was followed and what happens when it is broken. Adopt it as written, change the numbers in brackets to fit your volumes, and date each version.
| Clause | Requirement | Evidence kept | If breached |
|---|---|---|---|
| 1. Sealing | Holdout drawn and frozen before the first generation prompt, grounding index or seed set exists | Hash of the holdout file and the timestamp of sealing | Treat all results as development results, not validation |
| 2. Custodian | One named custodian controls item-level access; generator builders see scores only | Access list with roles, reviewed each release | Revoke access and record which items the person saw |
| 3. Separation | Seed examples, grounding sources and filter tuning use a separate real development split | Lineage file listing the source of every seed and grounding record | Rebuild the affected component without holdout records |
| 4. Touch log | Every scoring run and every item-level view is logged with purpose and person | Append-only log stored with the holdout | A missing log entry is treated as a touch |
| 5. Decision budget | Holdout used for at most [three] go or no-go decisions before rotation | Count of decision runs in the touch log | Rotate to a fresh sealed sample |
| 6. Strata | Exception types and rare outcomes kept at their real rate, with a minimum of [30] cases per reported stratum | Stratum table with counts and source period | Report the stratum as not measured, not as passing |
| 7. Memorization screen | Synthetic rows checked for exact copies of real rows and for being closer to training records than to holdout records | Screen output saved with the synthetic set version | Remove copied rows and rerun the comparison |
| 8. Negative controls | Each run includes controls that should fail, such as shuffled labels or a generator given no real grounding | Control scores beside the main result | If a control passes, the scoring pipeline is broken |
| 9. Privacy screen | Holdout cases de-identified and reviewed under the same rules as any shared real data | Review record and the rule set applied | Withdraw the case until it is reviewed |
| 10. Disclosure | Every reported result names the policy version, holdout period, strata, touch count and generation recipe | Derivation record attached to the result | Result is not cited outside the team |
How should the holdout be drawn so it reflects real work, exceptions included?
Draw it from the most recent complete period of real cases, after any data the generator or its grounding pool used, and keep exception types at the rate they actually occur. A random sample of an archive that also feeds the generator cannot be untouched, however carefully it is stored.
A time-based cut is the simplest defensible rule: everything before a date may feed generation, and a sample after that date becomes the holdout. It also tests whether the synthetic set generalizes to cases that arrived later, which is the condition a deployed model faces.
Size the holdout per stratum, not only in total. A holdout of a few thousand routine cases can still contain only a handful of the escalations or policy exceptions the model most needs to get right, and a handful cannot support a conclusion. Where a stratum is too small, say so in the report rather than folding it into the average.
Some comparisons also need the holdout to be large relative to the training data. The SDMetrics library, for example, warns in its overfitting protection metric when the real validation data is smaller than half the real training data used to fit the synthesizer, because the distance comparison becomes less reliable.
Keep a third real split for development. The generator builders need real cases to write seeds, check tone and tune filters, and if no development split exists they will reach for the holdout. A practical arrangement is a development pool from the earlier period, sized for prompt and filter work, and a sealed holdout from the later period that only the custodian and the scoring job can open.
Record how the sample was drawn in the stratum table: the source system, the date range, any records excluded and why, and the count per exception type. A reviewer who later asks why a stratum looks small should be able to answer from that table without reopening the holdout.
Which comparisons should the sealed cases score?
Score at least three things on the holdout: whether a model trained on synthetic data transfers to real cases, whether adding synthetic data to real data helps or hurts, and whether the synthetic set copied its source. Each answers a different question, and a set can pass one and fail another.
- Transfer: train on synthetic, test on the sealed real cases, and compare with a model trained on a real development set of similar size. This is the core validation.
- Augmentation: compare a model trained on real data alone with one trained on real plus synthetic data, both scored on the same sealed cases. SDMetrics implements this pattern in its data augmentation metrics, scoring both models on real validation data.
- Copying: check whether synthetic rows exactly match real rows, as the SDMetrics NewRowSynthesis metric does, and whether they sit closer to the records used for training than to the holdout, as its DCR overfitting protection metric does. Its documentation says the validation data should not have been used to train the synthesizer.
- Controls: run a model trained on label-shuffled synthetic data and one trained on generator output with no real grounding. Both should score worse; if they do not, the evaluation is measuring something other than learning.
- Per-stratum results: report every comparison for each exception type, not only overall, because an average can rise while a rare and costly case gets worse.
Reading per-exception results to find what the generator never learned
Read the per-stratum table first and the overall score last. The strata where the synthetic-trained model trails the real-trained model by the widest margin are the real conditions the synthetic mixture does not represent.
This is where synthetic data usually fails. Shumailov and colleagues reported in Nature that models trained on recursively generated data lose the tails of the original distribution first, so rare events disappear before common ones degrade. Business generators show a milder version of the same pattern: they reproduce the routine case well and under-produce the odd combination of policy, timing and customer history that makes an exception hard.
When a stratum fails, write down the missing condition in plain terms, such as a partial refund after a second complaint or a reschedule caused by a recalled part. That sentence becomes a generation brief or, more often, a request for more real cases of that type, because a generator that never saw the condition cannot invent it accurately.
What should the derivation record disclose with each result?
Every reported number should travel with a short derivation record: the generation recipe, the grounding sources, the holdout period and strata, the policy version and date, and how many times the holdout has been used. Without it, a reader cannot tell a validation result from a tuned one.
Regulators are moving toward the same distinction. The EU AI Act defines testing data as data used to provide an independent evaluation of an AI system before it is placed on the market, and it requires training, validation and testing data sets for high-risk systems to be relevant and sufficiently representative for the intended purpose. A holdout that fed the generator is not independent in that sense.
Mark the holdout files themselves too. The BIG-bench project puts a canary string in its task files to prevent benchmark tasks from leaking into web-scraped training data. A unique marker in each holdout export lets you search a training corpus or a generator's grounding pool for an accidental copy.
Keep the record short enough that people attach it. One block of plain fields works: synthetic set version, generator model and prompt version, grounding sources with their date ranges, holdout policy version, holdout period, number of sealed cases and strata, touch count to date, and the comparisons run. When the set is shared with another team or a vendor, the record goes with it, so a later reader can tell whether a score they are quoting came from a sealed holdout or from the development pool.
Worked example (illustrative): a dispatch assistant trained mostly on generated work orders
The figures below are invented to show the method; they are not results. A field service team wants an assistant that drafts reschedule notes. It has 18 months of real work orders and plans to train mostly on generated ones.
- Sealing: the last 3 months of work orders are cut off as the holdout pool, and a sample of 1,200 orders is sealed and hashed before any prompt is written. The earlier 15 months form the development pool for seeds and grounding.
- Strata: the 1,200 sealed orders hold 9 exception types; routine reschedules are 71 percent, and the rarest type, a reschedule after a recalled part, has 34 cases, just above the 30-case floor.
- First run: the synthetic-trained model trails the real-trained model by 3 points overall but by 22 points on recalled-part reschedules and 15 points on second-visit warranty disputes.
- Touch event: a reviewer finds that the grounding index was built from all 18 months. Under clause 3 the index is rebuilt from the development pool, and the first run is relabeled a development result.
- Second run: with the clean index, the overall gap widens to 5 points, a sign the first result was inflated. The two weak strata are written up as missing conditions, and the team requests more real orders of those types instead of generating more.
- Rotation: after the third decision run, the custodian seals a fresh sample from the following month and retires the first.
When the policy cannot be met: burned holdouts, thin strata and specialist review
Stop and narrow the claim when the holdout has been touched in a way you cannot undo, when an important stratum has too few cases to read, or when the result will drive a decision with legal, safety or financial consequences. Each needs a different response.
A burned holdout cannot be repaired by deleting the exposed items, because the generator may have generalized from them. Retire it, seal a new sample from a later period and say in the report which results came before the reset.
A thin stratum is reported as not measured. If that stratum matters most, get more real cases of that kind before validating anything, because no amount of synthetic data closes a gap you cannot see.
Bring in a domain validator when the exception categories themselves are uncertain. Someone who handles the work daily will spot a stratum that mixes two different situations, or a generated case that is plausible to an engineer but impossible under the business's own rules.
One exception to the time-based cut: when real cases are so scarce that holding any back leaves too little to ground the generator, use repeated rotation of small sealed samples and report the spread across rotations rather than a single score.
Where licensed real records fit in a holdout program
Holdouts consume real data: each rotation needs a fresh sample that was never near the generator, and the thin strata need more real cases than an internal archive often holds. Synthetic data built from identified records is still treated as a de-identification technique in NIST SP 800-188, so the real side needs privacy review whether it seeds a generator or serves as a holdout.
Teams short of real cases for a holdout can describe what they need to SourceX, which sources datasets from U.S. businesses on request rather than from stock; a request there records the scope, volume, format, timing and licensing terms the holdout needs. The supplying business approves scope and terms and signs before delivery, and that agreement is the place to state whether records may be used for evaluation only, for generation, or both.
Frequently asked questions
Can the same real records seed the generator and validate it?
No, not if the result is meant to show how the synthetic set performs on new real work. Records that seeded or grounded the generator can only give a development result. Split the real data first and keep the validation share out of every generation step.
How many times can a holdout be used before it stops being untouched?
There is no universal number, which is why the policy sets a decision budget and logs each run. Every result used to change the generator fits it a little more to the holdout. When the budget is spent, seal a new sample from a later period.
Is a high fidelity or similarity score enough to validate synthetic data?
No. Similarity scores compare the synthetic set with real data in aggregate; they do not show whether a model trained on it handles real exceptions. Use them alongside a train on synthetic, test on real comparison on sealed cases.
What is a negative control in synthetic data validation?
A negative control is a run that should fail, such as a model trained on synthetic data with shuffled labels. If the control scores well on the holdout, the evaluation pipeline has a leak or a bug, and the main result should not be trusted until it is fixed.
Does the holdout need de-identification if it never leaves the team?
It needs the same privacy review as any real records the team holds, because people and tools still read it. If holdout cases come from another company or will be shared with vendors or contractors, de-identification and the licensing terms decide who may see them.
Sources
- A 2017 paper on recurrent conditional GANs for medical time series describes evaluating generated time series by training on a synthetic labelled set and testing on real data, and the reverse. Source
- SDMetrics' DCROverfittingProtection compares how close synthetic rows are to the real training data versus a real holdout, states that the validation data should not have been used to train the synthesizer, and warns when the validation data is smaller than half the training data. Source
- SDMetrics' NewRowSynthesis metric measures whether each synthetic row is new or exactly matches a row in the real data. Source
- SDMetrics' data augmentation metrics score a classifier trained on real data alone and one trained on real plus synthetic data on the same real validation data. Source
- BIG-bench task files contain a canary string intended to prevent benchmark tasks from leaking into web-scraped training data. Source
- The EU AI Act defines testing data as data used to provide an independent evaluation of an AI system, and Article 10(3) requires training, validation and testing data sets for high-risk systems to be relevant and sufficiently representative for the intended purpose. Source
- Shumailov et al. report in Nature that training on recursively generated data causes irreversible defects in which the tails of the original content distribution disappear. Source
- NIST SP 800-188 lists generating synthetic data with models among de-identification techniques and publishing synthetic data based on identified data among data-sharing models. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.