Skip to content

AI uses for records

Synthetic data grounding audit: what real evidence grounds a business dataset?

By SourceX Editorial · Updated

10 sections

Short answer

A synthetic business dataset should be grounded in real evidence: de-identified source records, policies dated to the scenario, and patterns seen in real records. A grounding audit samples generated examples, traces each to its source, checks policy dates and business rules, compares it with an untouched real holdout, screens for near copies and labels anything with no trace as an unsupported scenario. Disclose the counts.

Checklist

Synthetic data grounding

This page covers synthetic data grounding audit: what real evidence grounds a business dataset.Illustrative

Key takeaways

  • Every generated example should trace to a source record, a dated rule or a real pattern; anything else is an unsupported scenario.
  • Distribution and utility scores do not catch examples that cite stale policies or invented codes; a per-example audit does.
  • Classify failures by cause, because stale rules, invented values and impossible combinations each need a different fix.
  • Keep a real holdout away from the generator and plant negative controls to test the auditors.
  • Disclose generator, grounding sources, policy dates, verdict counts and control catch rate with the dataset.

What real evidence should ground a synthetic business dataset?

Every generated example should trace to at least one piece of real evidence: a de-identified source record it was derived from, a policy or rule in force on a stated date, or a documented pattern seen in real records. Examples with no such trace are unsupported scenarios. They are not automatically wrong, but they must be labeled, counted and tested against an untouched set of real cases before anyone trains on them.

A grounding audit makes that trace explicit for a sample of the synthetic set. It records where each example came from, which rules it claims to follow, whether those rules were current, and whether real held-out records contain anything like it. The result is a disclosure buyers can read and a list of scenarios the generator invented.

This matters most for business data, where the content is procedural: refund rules, approval limits, claim codes, service-level terms. A fluent example that quotes last year's policy or a code the business never used teaches a model something false with full confidence.

How does a grounding audit differ from a synthetic quality score?

Quality scores compare distributions; a grounding audit checks individual examples against evidence. SDMetrics, the open-source evaluation library from the Synthetic Data Vault project, compares synthetic data to the real data it imitates, with reports on column shapes and pair trends and single metrics such as boundary adherence, whether synthetic values stay within the real minimum and maximum, and new row synthesis, whether rows are new rather than copies of real ones.

Utility tests go one step further. Esteban, Hyland and Rätsch proposed training a model on synthetic data and testing it on real data, an approach known as train on synthetic, test on real (TSTR). That shows whether the synthetic set is useful in aggregate, but a high score can coexist with many examples that cite an expired rule or describe a case the business never handles.

The audit sits beside these measures rather than replacing them. Run distribution and utility checks on the whole set, and run the grounding audit on a stratified sample, oversampling rare categories where generators invent most.

Which kinds of real evidence can ground a generated example?

Grounding comes in four strengths, and the audit should name which one applies to each example. Stronger grounding does not remove the need for a privacy screen; an example derived from a single real record can leak that record.

Which kinds of real evidence can ground a generated example?
Evidence classWhat it provesBusiness exampleWeakness
Derived from a source recordA real case of this kind existedParaphrased Zendesk ticket with names and amounts removedCan reproduce identifying details; needs a privacy check
Rule or policy groundedThe example follows a documented rule in force on a dateReturn window from the policy version dated in the exampleProves consistency with the rule, not that the case happens
Pattern groundedThe combination of fields appears in real aggregatesClaim type and denial reason pair seen in the real logReal frequency may differ sharply from synthetic frequency
Expert attestedA domain expert says the case is realisticSenior agent confirms an unusual escalation pathOpinion, not record; needs a named reviewer and date
NoneNothing beyond the generatorScenario the prompt asked for with no matching record or ruleUnsupported; label, test and decide case by case

What should the source grounding and unsupported scenario audit include?

Audit a sample of examples, one row each. The fields below make the audit repeatable by a second reviewer and give a buyer the evidence behind every verdict. Entries are illustrative.

At a glance

What should the source grounding and unsupported scenario audit include

What should the source grounding and unsupported scenario audit include?
FieldWhat to recordWhere it comes fromIllustrative entry
Example and batch IDsSynthetic example ID, generation batch, generator and prompt versionGeneration logS-2291, batch 7, prompt template v3
Generation recipeSeed records or documents supplied, decoding settings, filters appliedPipeline configurationSeeded with 3 de-identified tickets; duplicate filter on
Grounding sourceThe record, rule or pattern the example traces toSeed index, policy library, real aggregatesRefund policy version of 1 March, clause 4
Evidence classDerived, rule, pattern, expert or noneAuditor judgmentRule grounded
Policy date checkWhether every rule cited was in force for the example's scenario datePolicy version historyFails: cites a 30-day window replaced by 14 days
Rule consistencyWhether amounts, codes and steps obey the business rulesRule tableApproval limit respected; reason code valid
Holdout matchNearest similar case in the untouched real holdout, if anySimilarity search on the holdoutTwo close real cases found
Privacy screenWhether the example reproduces a real person, account or unusual detailNearest-record check on seed dataPass; no near copy
VerdictGrounded, grounded with fix, unsupported but plausible, or rejectAuditor, with second review on disagreementsGrounded with fix: update return window

How should unsupported scenarios be classified and handled?

Classify each unsupported or failing example by why it fails, because the fix differs. A stale rule is a generator input problem; an impossible combination is a constraint problem; a plausible but unseen case is a coverage question for the domain expert.

How should unsupported scenarios be classified and handled?
CategoryHow to detect itDefault verdictAction
Stale rulePolicy date check failsReject or fixUpdate the policy library given to the generator
Invented code or field valueValue absent from the real code listRejectConstrain generation to the real code list
Impossible combinationViolates a rule such as an approval limit or a date orderRejectAdd the rule as a hard filter
Plausible but unseenNo holdout match; expert finds it realisticKeep, labeled unsupportedRecord the expert, the date and the reason
Over-represented edge caseSynthetic share far above real shareKeep, reweightRebalance the mixture to the real frequency
Near copy of a real recordPrivacy screen flags a close matchRejectStrengthen de-identification and dedupe against seeds

Which real exception types never appear in the generated set?

Generated sets tend to miss the long tail. Shumailov and colleagues showed that training models on model-generated content causes the tails of the original distribution to disappear, an effect they call model collapse. A single generation pass can narrow in a similar way when prompts ask for typical cases, which is a separate risk from the one the paper measured.

Compare categories, not just averages. List the exception types in the real holdout, such as partial refunds, disputes that span two policies, cases reopened after closure, accounts with unusual histories, and records with missing fields, and count how many synthetic examples fall into each. Categories with real cases but no synthetic ones are gaps; categories with synthetic cases but no real ones are unsupported scenarios to send through the audit.

Pipelines that bootstrap from a model's own output need the same check. Self-Instruct, for example, generates instructions and examples from a language model and filters invalid or near-duplicate generations before fine-tuning. Filtering removes bad and repeated items; it does not add the real exceptions the seeds never contained.

Keeping an untouched real holdout and planting negative controls

Keep a set of real cases that never reaches the generator, its prompts or its filters, and use it for both the holdout match and the final utility test. If seed records also appear in the evaluation set, the synthetic data will look better grounded than it is.

Plant negative controls to test the audit itself. Before the audit, insert a handful of examples that are deliberately ungrounded, such as one citing a retired policy, one using a code that does not exist and one with an impossible date order, without telling the auditors which ones. An audit that does not catch its own controls cannot be trusted to catch the generator's mistakes, and the catch rate belongs in the disclosure.

Worked example: auditing a synthetic refund-dispute set for a support agent

Every number in this scenario is illustrative and invented for teaching. A team builds a synthetic set of refund-dispute conversations to fine-tune a support agent, seeded with de-identified tickets and the company's refund policy library, and audits a stratified sample of 300 examples plus 6 planted controls.

The audit finds 211 examples grounded, 38 grounded but needing a fix because they cite a return window replaced earlier in the year, 19 using reason codes that do not exist, 9 impossible combinations where refunds exceed the original charge, and 23 plausible but unseen cases. Auditors catch 5 of the 6 controls; the missed one, a retired loyalty rule, leads to adding loyalty terms to the policy check. The holdout comparison shows no synthetic examples of disputes spanning a policy change, a common real case.

The exception: a rare chargeback pattern has only two real cases, too few to seed safely. A senior agent attests that the generated variants are realistic, so they are kept, labeled expert attested with the reviewer and date, and excluded from the evaluation set so the agent is never scored on its own invented cases.

What to disclose, and when to stop using the generated set

Disclose the derivation with the dataset. Gebru and colleagues' Datasheets for Datasets proposes that every dataset carry documentation of its motivation, composition and collection process; for synthetic data that means the generator, the grounding sources, the policy dates, the audit sample, the verdict counts, the control catch rate and the share of unsupported scenarios.

At a glance

What to disclose, and when to stop using the generated set

NIST's Generative AI Profile describes confabulation as confidently stated but erroneous or false content. A synthetic training set with many unsupported examples passes that risk to every model trained on it, which is why the audit results belong next to the data rather than in an internal ticket.

  • Stop and regenerate when stale rules or invented codes are common in the sample; the generator's inputs are wrong.
  • Narrow the scope to the categories that audit cleanly when only some scenario types are well grounded.
  • Request specialist review from a domain validator when unsupported but plausible cases make up a large share.
  • Request privacy review when near copies appear, or when seeds include personal or sensitive records.
  • Return to real data when the holdout shows exception types the generator cannot produce.

Sourcing the real records a grounding audit depends on

A grounding audit is only as good as the real evidence behind it: de-identified source records, dated policy versions, real code lists and an untouched holdout that reflects how often exceptions actually occur. Teams that lack these often generate from public examples and cannot tell which scenarios their business counterpart would ever see.

SourceX can be asked to locate U.S. businesses whose records supply grounding evidence of this kind; each dataset is sourced for the request, and the form records scope, volume, format, timing and licensing needs. The records arrive under an agreement the business has approved and signed, which is the point to state whether they may seed a generator, serve as a holdout or both.

Frequently asked questions

What is a synthetic data grounding audit?

It is a per-example review of a sample of generated data that records what real evidence each example traces to, whether its rules were current, whether similar real cases exist and whether it copies a real record. It ends with verdict counts and a list of unsupported scenarios.

How many synthetic examples should be audited?

Enough to estimate the share of unsupported and failing examples in each scenario category with useful precision, which usually means a stratified sample that oversamples rare categories. Report the sample design with the results so readers can judge the uncertainty.

Is an unsupported scenario always bad?

No. Some plausible cases are simply rare in the real records. Keep them only with a named expert's attestation, label them, and keep them out of the evaluation set so a model is not scored on cases nobody has observed.

Can the seed data double as the test set?

It should not. If seed records or their near copies appear in the evaluation set, the synthetic data will look better than it is; keep a separate real holdout that never reaches the generator.

Who should run the audit?

A data team for the tracing and screens, plus a domain validator who knows the business rules and exceptions. A second reviewer should settle disagreements, and the planted controls show how reliable the reviewers were.

Sources

  • SDMetrics evaluates synthetic data against the real data it imitates, including boundary adherence and whether rows are new rather than copies. Source
  • Esteban, Hyland and Rätsch evaluated synthetic data by training on synthetic data and testing on real data. Source
  • Training on model-generated content causes the tails of the original distribution to disappear, which the authors call model collapse. Source
  • Self-Instruct generates instructions and examples from a language model and filters invalid or near-duplicate generations before fine-tuning. Source
  • Datasheets for Datasets proposes documenting each dataset's motivation, composition and collection process. Source
  • NIST AI 600-1 describes confabulation as the production of confidently stated but erroneous or false content. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies