AI uses for records
Synthetic data grounding audit: what real evidence grounds a business dataset?
By SourceX Editorial · Updated
10 sections
Short answer
A synthetic business dataset should be grounded in real evidence: de-identified source records, policies dated to the scenario, and patterns seen in real records. A grounding audit samples generated examples, traces each to its source, checks policy dates and business rules, compares it with an untouched real holdout, screens for near copies and labels anything with no trace as an unsupported scenario. Disclose the counts.
Checklist
Synthetic data grounding
Key takeaways
- Every generated example should trace to a source record, a dated rule or a real pattern; anything else is an unsupported scenario.
- Distribution and utility scores do not catch examples that cite stale policies or invented codes; a per-example audit does.
- Classify failures by cause, because stale rules, invented values and impossible combinations each need a different fix.
- Keep a real holdout away from the generator and plant negative controls to test the auditors.
- Disclose generator, grounding sources, policy dates, verdict counts and control catch rate with the dataset.
What real evidence should ground a synthetic business dataset?
Every generated example should trace to at least one piece of real evidence: a de-identified source record it was derived from, a policy or rule in force on a stated date, or a documented pattern seen in real records. Examples with no such trace are unsupported scenarios. They are not automatically wrong, but they must be labeled, counted and tested against an untouched set of real cases before anyone trains on them.
A grounding audit makes that trace explicit for a sample of the synthetic set. It records where each example came from, which rules it claims to follow, whether those rules were current, and whether real held-out records contain anything like it. The result is a disclosure buyers can read and a list of scenarios the generator invented.
This matters most for business data, where the content is procedural: refund rules, approval limits, claim codes, service-level terms. A fluent example that quotes last year's policy or a code the business never used teaches a model something false with full confidence.
How does a grounding audit differ from a synthetic quality score?
Quality scores compare distributions; a grounding audit checks individual examples against evidence. SDMetrics, the open-source evaluation library from the Synthetic Data Vault project, compares synthetic data to the real data it imitates, with reports on column shapes and pair trends and single metrics such as boundary adherence, whether synthetic values stay within the real minimum and maximum, and new row synthesis, whether rows are new rather than copies of real ones.
Utility tests go one step further. Esteban, Hyland and Rätsch proposed training a model on synthetic data and testing it on real data, an approach known as train on synthetic, test on real (TSTR). That shows whether the synthetic set is useful in aggregate, but a high score can coexist with many examples that cite an expired rule or describe a case the business never handles.
The audit sits beside these measures rather than replacing them. Run distribution and utility checks on the whole set, and run the grounding audit on a stratified sample, oversampling rare categories where generators invent most.
Which kinds of real evidence can ground a generated example?
Grounding comes in four strengths, and the audit should name which one applies to each example. Stronger grounding does not remove the need for a privacy screen; an example derived from a single real record can leak that record.
| Evidence class | What it proves | Business example | Weakness |
|---|---|---|---|
| Derived from a source record | A real case of this kind existed | Paraphrased Zendesk ticket with names and amounts removed | Can reproduce identifying details; needs a privacy check |
| Rule or policy grounded | The example follows a documented rule in force on a date | Return window from the policy version dated in the example | Proves consistency with the rule, not that the case happens |
| Pattern grounded | The combination of fields appears in real aggregates | Claim type and denial reason pair seen in the real log | Real frequency may differ sharply from synthetic frequency |
| Expert attested | A domain expert says the case is realistic | Senior agent confirms an unusual escalation path | Opinion, not record; needs a named reviewer and date |
| None | Nothing beyond the generator | Scenario the prompt asked for with no matching record or rule | Unsupported; label, test and decide case by case |
What should the source grounding and unsupported scenario audit include?
Audit a sample of examples, one row each. The fields below make the audit repeatable by a second reviewer and give a buyer the evidence behind every verdict. Entries are illustrative.
At a glance
What should the source grounding and unsupported scenario audit include
| Field | What to record | Where it comes from | Illustrative entry |
|---|---|---|---|
| Example and batch IDs | Synthetic example ID, generation batch, generator and prompt version | Generation log | S-2291, batch 7, prompt template v3 |
| Generation recipe | Seed records or documents supplied, decoding settings, filters applied | Pipeline configuration | Seeded with 3 de-identified tickets; duplicate filter on |
| Grounding source | The record, rule or pattern the example traces to | Seed index, policy library, real aggregates | Refund policy version of 1 March, clause 4 |
| Evidence class | Derived, rule, pattern, expert or none | Auditor judgment | Rule grounded |
| Policy date check | Whether every rule cited was in force for the example's scenario date | Policy version history | Fails: cites a 30-day window replaced by 14 days |
| Rule consistency | Whether amounts, codes and steps obey the business rules | Rule table | Approval limit respected; reason code valid |
| Holdout match | Nearest similar case in the untouched real holdout, if any | Similarity search on the holdout | Two close real cases found |
| Privacy screen | Whether the example reproduces a real person, account or unusual detail | Nearest-record check on seed data | Pass; no near copy |
| Verdict | Grounded, grounded with fix, unsupported but plausible, or reject | Auditor, with second review on disagreements | Grounded with fix: update return window |
How should unsupported scenarios be classified and handled?
Classify each unsupported or failing example by why it fails, because the fix differs. A stale rule is a generator input problem; an impossible combination is a constraint problem; a plausible but unseen case is a coverage question for the domain expert.
| Category | How to detect it | Default verdict | Action |
|---|---|---|---|
| Stale rule | Policy date check fails | Reject or fix | Update the policy library given to the generator |
| Invented code or field value | Value absent from the real code list | Reject | Constrain generation to the real code list |
| Impossible combination | Violates a rule such as an approval limit or a date order | Reject | Add the rule as a hard filter |
| Plausible but unseen | No holdout match; expert finds it realistic | Keep, labeled unsupported | Record the expert, the date and the reason |
| Over-represented edge case | Synthetic share far above real share | Keep, reweight | Rebalance the mixture to the real frequency |
| Near copy of a real record | Privacy screen flags a close match | Reject | Strengthen de-identification and dedupe against seeds |
Which real exception types never appear in the generated set?
Generated sets tend to miss the long tail. Shumailov and colleagues showed that training models on model-generated content causes the tails of the original distribution to disappear, an effect they call model collapse. A single generation pass can narrow in a similar way when prompts ask for typical cases, which is a separate risk from the one the paper measured.
Compare categories, not just averages. List the exception types in the real holdout, such as partial refunds, disputes that span two policies, cases reopened after closure, accounts with unusual histories, and records with missing fields, and count how many synthetic examples fall into each. Categories with real cases but no synthetic ones are gaps; categories with synthetic cases but no real ones are unsupported scenarios to send through the audit.
Pipelines that bootstrap from a model's own output need the same check. Self-Instruct, for example, generates instructions and examples from a language model and filters invalid or near-duplicate generations before fine-tuning. Filtering removes bad and repeated items; it does not add the real exceptions the seeds never contained.
Keeping an untouched real holdout and planting negative controls
Keep a set of real cases that never reaches the generator, its prompts or its filters, and use it for both the holdout match and the final utility test. If seed records also appear in the evaluation set, the synthetic data will look better grounded than it is.
Plant negative controls to test the audit itself. Before the audit, insert a handful of examples that are deliberately ungrounded, such as one citing a retired policy, one using a code that does not exist and one with an impossible date order, without telling the auditors which ones. An audit that does not catch its own controls cannot be trusted to catch the generator's mistakes, and the catch rate belongs in the disclosure.
Worked example: auditing a synthetic refund-dispute set for a support agent
Every number in this scenario is illustrative and invented for teaching. A team builds a synthetic set of refund-dispute conversations to fine-tune a support agent, seeded with de-identified tickets and the company's refund policy library, and audits a stratified sample of 300 examples plus 6 planted controls.
The audit finds 211 examples grounded, 38 grounded but needing a fix because they cite a return window replaced earlier in the year, 19 using reason codes that do not exist, 9 impossible combinations where refunds exceed the original charge, and 23 plausible but unseen cases. Auditors catch 5 of the 6 controls; the missed one, a retired loyalty rule, leads to adding loyalty terms to the policy check. The holdout comparison shows no synthetic examples of disputes spanning a policy change, a common real case.
The exception: a rare chargeback pattern has only two real cases, too few to seed safely. A senior agent attests that the generated variants are realistic, so they are kept, labeled expert attested with the reviewer and date, and excluded from the evaluation set so the agent is never scored on its own invented cases.
What to disclose, and when to stop using the generated set
Disclose the derivation with the dataset. Gebru and colleagues' Datasheets for Datasets proposes that every dataset carry documentation of its motivation, composition and collection process; for synthetic data that means the generator, the grounding sources, the policy dates, the audit sample, the verdict counts, the control catch rate and the share of unsupported scenarios.
At a glance
What to disclose, and when to stop using the generated set
NIST's Generative AI Profile describes confabulation as confidently stated but erroneous or false content. A synthetic training set with many unsupported examples passes that risk to every model trained on it, which is why the audit results belong next to the data rather than in an internal ticket.
- Stop and regenerate when stale rules or invented codes are common in the sample; the generator's inputs are wrong.
- Narrow the scope to the categories that audit cleanly when only some scenario types are well grounded.
- Request specialist review from a domain validator when unsupported but plausible cases make up a large share.
- Request privacy review when near copies appear, or when seeds include personal or sensitive records.
- Return to real data when the holdout shows exception types the generator cannot produce.
Sourcing the real records a grounding audit depends on
A grounding audit is only as good as the real evidence behind it: de-identified source records, dated policy versions, real code lists and an untouched holdout that reflects how often exceptions actually occur. Teams that lack these often generate from public examples and cannot tell which scenarios their business counterpart would ever see.
SourceX can be asked to locate U.S. businesses whose records supply grounding evidence of this kind; each dataset is sourced for the request, and the form records scope, volume, format, timing and licensing needs. The records arrive under an agreement the business has approved and signed, which is the point to state whether they may seed a generator, serve as a holdout or both.
Frequently asked questions
What is a synthetic data grounding audit?
It is a per-example review of a sample of generated data that records what real evidence each example traces to, whether its rules were current, whether similar real cases exist and whether it copies a real record. It ends with verdict counts and a list of unsupported scenarios.
How many synthetic examples should be audited?
Enough to estimate the share of unsupported and failing examples in each scenario category with useful precision, which usually means a stratified sample that oversamples rare categories. Report the sample design with the results so readers can judge the uncertainty.
Is an unsupported scenario always bad?
No. Some plausible cases are simply rare in the real records. Keep them only with a named expert's attestation, label them, and keep them out of the evaluation set so a model is not scored on cases nobody has observed.
Can the seed data double as the test set?
It should not. If seed records or their near copies appear in the evaluation set, the synthetic data will look better than it is; keep a separate real holdout that never reaches the generator.
Who should run the audit?
A data team for the tracing and screens, plus a domain validator who knows the business rules and exceptions. A second reviewer should settle disagreements, and the planted controls show how reliable the reviewers were.
Sources
- SDMetrics evaluates synthetic data against the real data it imitates, including boundary adherence and whether rows are new rather than copies. Source
- Esteban, Hyland and Rätsch evaluated synthetic data by training on synthetic data and testing on real data. Source
- Training on model-generated content causes the tails of the original distribution to disappear, which the authors call model collapse. Source
- Self-Instruct generates instructions and examples from a language model and filters invalid or near-duplicate generations before fine-tuning. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition and collection process. Source
- NIST AI 600-1 describes confabulation as the production of confidently stated but erroneous or false content. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.