AI uses for records
Synthetic data exception preservation: how to find what it dropped
By SourceX Editorial · Updated
10 sections
Short answer
To detect business exceptions missing from synthetic training data, write each exception as an executable rule, count it in the synthetic set and in an untouched real holdout, compare the rates with exact intervals, and check that every synthetic exception obeys the rule that defines it. Classes that are absent, rarer or internally inconsistent go into a gap report, alongside how the synthetic data was grounded, generated and selected.
Agent run
Synthetic data exception
Key takeaways
- Aggregate fidelity scores can look good while rare business exceptions are missing, because they average over columns or set outliers aside.
- An exception must be defined as a rule over fields, such as a refund above the policy limit with an approver recorded, before it can be counted.
- Compare exception rates against a real holdout that never touched the generator, using exact binomial intervals for rare classes.
- Count is not enough: every synthetic exception should also satisfy the rule that defines it, or it teaches the wrong behavior.
- A class too rare in the real holdout to measure is a finding in itself; it calls for more real records, not a pass.
- Ask synthetic suppliers to disclose grounding records, generation recipe, selection filters and the policy date their rules reflect.
How do you detect business exceptions missing from synthetic training data?
Define each exception as a rule over fields, count it in both the synthetic set and an untouched real holdout, compare the rates with intervals, and test whether each synthetic exception is internally consistent. Anything absent, under-represented or inconsistent is a gap, and the gap report is what you accept or reject the data on.
Business exceptions are the cases where normal processing stops: a refund above the policy limit, an invoice that fails matching, a shipment split across warehouses, a ticket reopened after closure. They are often the cases a model or agent is bought to handle, and they are exactly where generated data tends to be thin.
The method works on the data itself, before any model is trained. That makes it cheap to run on a vendor sample, and it gives a buyer specific, checkable reasons to accept, renegotiate or reject a synthetic or mixed dataset.
Why does synthetic data smooth away exceptions?
Generators learn the bulk of a distribution more easily than its tails, and common quality metrics are built to summarize the bulk. Both push in the same direction: rare combinations of fields fade, and the scorecard does not show it.
Research on recursive training shows the mechanism at its extreme. Shumailov and colleagues, writing in Nature, found that training on model-generated content causes defects in which tails of the original content distribution disappear. A later theoretical study reports that this collapse can persist even when real and synthetic data are mixed, as long as the synthetic fraction is not zero. Your vendor's generator may not be recursive, but the lesson about tails carries over.
Standard metrics can hide the loss. The alpha-precision and beta-recall metrics proposed by Alaa and colleagues measure fidelity and diversity at sample level, and by design they set outliers aside when doing so. The SDMetrics quality report scores Column Shapes and Column Pair Trends as averages across columns and pairs. A refund-override pattern that lives in three fields and one percent of rows barely moves either kind of score.
What counts as a business exception in operational records?
A business exception is a record, or sequence of records, where a stated rule was triggered and something other than the default happened. To be testable it needs a predicate: fields and values that decide, without judgment, whether a row is in the class. The table gives illustrative classes from common systems; replace the thresholds with your own policy values.
| Exception class (illustrative) | Typical system | Rule that defines it | Consistency check |
|---|---|---|---|
| Refund above policy limit | Support desk or order system | refund_amount greater than policy_limit | approver_id present and approval timestamp before refund timestamp |
| Invoice failed three-way match | ERP accounts payable | invoice quantity or price differs from PO or receipt beyond tolerance | hold status set, then a release or credit memo follows |
| Partial shipment with backorder | Order management or WMS | shipped_qty less than ordered_qty on a line | backorder line exists and later ship date is after first |
| SLA breach with escalation | Ticketing system | first response time greater than SLA target | escalation event exists and follows the breach |
| Ticket reopened after closure | Ticketing system | status changes from closed to open | new customer message or agent note between close and reopen |
| Duplicate customer merged | CRM | merge event links two customer ids | surviving id keeps history of both; merged id stops receiving activity |
| Chargeback after refund | Payments and order system | chargeback event on an order already refunded | dispute outcome recorded; no second refund issued |
How do you run the exception frequency comparison?
Run seven steps in order, and freeze the real holdout before anything else, because a holdout that informed the generator cannot reveal what the generator missed. The steps take a data engineer and a process owner, not a research team.
Use exact intervals for rare classes. NIST's engineering statistics handbook notes that when the sample is small, confidence limits approximated with the normal distribution may not be accurate enough, and it gives exact binomial limits instead. Exception classes are small samples by definition, so compute exact (Clopper-Pearson) intervals for each class rate in both sets.
- 1. Freeze a real holdout from the same systems and period, never used for grounding, prompting, fine-tuning or filtering the generator. Record its date range and policy version.
- 2. Write each exception class as a predicate over fields, reviewed by the process owner who knows the policy. Version the predicates.
- 3. Count each class in the holdout and in the synthetic set, as rows and, for sequence classes, as cases (ticket, order, invoice).
- 4. Compute each class rate with an exact binomial interval in both sets.
- 5. Run the consistency check on every synthetic row in each class: required fields present, events in a valid order, values within system limits.
- 6. Check co-occurrence for pairs of classes that occur together in the holdout, such as an SLA breach followed by a refund override.
- 7. Write the gap report and attach the generation disclosure from the supplier.
What should the exception gap report include?
The report needs one row per exception class with both rates, their intervals, a consistency pass rate and a verdict, plus a header that states the holdout's date range, the predicate version and the generator disclosure. A reviewer should be able to reproduce every number from the predicates and the two datasets.
Use the fields and flag rules below as is. Thresholds are starting points for a technical reviewer, not standards; set them before you look at the results so the verdicts cannot drift toward acceptance.
| Report field | Definition | How to compute | Flag rule |
|---|---|---|---|
| Real rate and interval | Share of holdout rows or cases in the class | Count over total, exact binomial 95% interval | Inconclusive if fewer than about 10 real occurrences |
| Synthetic rate and interval | Share of synthetic rows or cases in the class | Same method on the synthetic set | Missing if zero synthetic occurrences |
| Rate ratio | Synthetic rate divided by real rate | Point ratio; note whether the two intervals overlap | Under-represented if intervals do not overlap and synthetic is lower |
| Consistency pass rate | Share of synthetic class rows passing the consistency check | Passing rows over class rows | Distorted if below the threshold set in advance |
| Co-occurrence gap | Class pairs common in the holdout but absent together in synthetic data | Pair counts in both sets | Flag any pair present in the holdout and absent in synthetic data |
| Verdict | Preserved, under-represented, missing, distorted or inconclusive | Apply the flag rules in order | Any missing or distorted class blocks acceptance for that use |
Worked example: refunds and escalations in support tickets (illustrative)
Every number here is invented to show the method; it is not a study result. A buyer evaluates a vendor's synthetic support-ticket set of 100,000 tickets, generated from a seed the vendor describes, against a frozen real holdout of 20,000 tickets from one company's help desk covering two years.
In the holdout, refunds above the policy limit appear in 180 tickets; in the synthetic set, in 310. The rates are about 0.9% and 0.3%, the exact intervals do not overlap, and the class is under-represented. Worse, only 40% of the synthetic overrides carry an approver, against every one in the holdout, so the class is also distorted: a model trained on it would learn that large refunds need no approval.
Tickets reopened after closure appear in 1,150 holdout tickets and at a similar rate in the synthetic set, with a high consistency pass rate; that class is preserved. Chargebacks after a refund appear 46 times in the holdout and never in the synthetic set: missing.
The exception that matters most is the one the method cannot decide. Refunds issued in a currency other than the account currency appear only four times in the real holdout. The interval is too wide to judge any synthetic rate against, so the verdict is inconclusive, and the right action is to source more real records of that class, not to accept the synthetic version on trust.
What should a synthetic data supplier disclose?
Ask for the derivation: which real records grounded the generator, how examples were generated and filtered, and which policy date the rules reflect. Without that, a gap can be a generator limit, a selection filter or an outdated rule, and each needs a different fix.
At a glance
What should a synthetic data supplier disclose
- Real holdout: confirmation that the buyer's or a held-back real set never touched generation, with its date range
- Generation recipe: model or method, prompts or conditioning, and any class balancing applied
- Grounding sources: which real records seeded the generator and under what rights
- Selection filters: what was dropped after generation, such as outliers, failed validations or near-duplicates
- Policy date: the version of business rules the synthetic records follow, and whether it changes over the series
- Rule consistency: the supplier's own constraint checks and their pass rates
- Privacy screen: how memorized or near-copied real records were detected and removed
- Negative controls: deliberately invalid examples, if any, and how they are labeled
Which exception shapes do generators most often drop?
The classes most often missed share a structure: they need several fields, several records or a change over time to exist. Single-field rarities are easier for a generator to reproduce than multi-step ones, so check these first.
Sequence exceptions, such as reopen after close or chargeback after refund, need ordered events across a case. Cross-record exceptions, such as duplicate customers or split shipments, need two or more linked rows. Policy-change exceptions need records before and after a rule changed, which a generator conditioned on one policy cannot produce. Timing exceptions, such as a dispute filed months after an order, need long gaps that short generation windows rarely include.
To test these classes, extend the predicates beyond single rows. A sequence predicate checks event order within a case, a cross-record predicate joins on the shared key, and a policy-change predicate splits the holdout at the rule's effective date and compares each side separately. If the synthetic set has no case ids, timestamps or join keys, these classes cannot be tested at all, and that absence belongs in the gap report as a finding rather than being left out.
Mixtures need the same check per source. When a supplier blends real and generated records, ask for a flag on each row showing its origin, and run the comparison on the generated rows alone as well as on the blend. A blend can look preserved overall while every generated row in a class is inconsistent.
Which gap-report results should block, narrow or escalate acceptance?
Stop accepting the set for a use when any class that use depends on is missing or distorted; narrow the use when gaps sit in classes it does not need; request specialist review when results are inconclusive or when real exceptions are also privacy outliers.
That last case is common. Stadler and colleagues found that synthetic data either did not prevent inference attacks or did not retain utility, and that outlier records largely remained vulnerable. Rare exceptions are often outliers, so a synthetic set that preserves them faithfully may also expose the real cases behind them. A privacy reviewer should check those classes specifically.
Bring in a domain validator when predicates are disputed. If the process owner and the supplier disagree about what counts as an override or a valid escalation, settle the rule in writing before rerunning the comparison.
Where do real records for the missing exception classes come from?
When the gap report shows missing or inconclusive classes, the fix is usually more real records of those classes. Because SourceX sources datasets for AI teams from U.S. businesses on request, with no standing inventory, a request can list the exception classes and predicates straight from your gap report.
On the supply side, buyers usually want two or more years of consistent history and tens of thousands of records from one system, but SourceX's general guide leaves room for smaller sets when the work is rare, which is where exception-heavy slices often sit; each is assessed deal by deal.
Frequently asked questions
Is synthetic data better than real data for rare cases?
Not by default. Generators tend to reproduce common patterns more faithfully than rare ones, and averaged quality scores can hide that. Synthetic data can help when it is checked class by class against real records and its gaps are filled with real examples.
What are the disadvantages of using synthetic data for business processes?
The main ones are missing or distorted exceptions, rules that reflect one policy date only, and inconsistent multi-step sequences. Rare real cases can also be privacy outliers, so preserving them faithfully can carry re-identification risk.
How large does the real holdout need to be?
Large enough that each exception class you care about appears often enough to give a usable interval; around ten or more occurrences per class is a practical floor. If a class is rarer than that, the holdout cannot judge it and you need more real records of that class.
Can I use the same real data to ground the generator and to test it?
No. A holdout that informed the generator, its prompts or its filters cannot show what the generator missed. Freeze the holdout first and record that it was never used for generation.
Do column-level quality scores catch missing exceptions?
Usually not. Scores such as column shape and column pair similarity are averaged across columns, and some sample-level metrics set outliers aside by design. An exception defined over three fields and one percent of rows barely moves them.
Sources
- Shumailov and colleagues found that training on model-generated content causes defects in which tails of the original content distribution disappear. Source
- A theoretical study reports that model collapse can persist when real and synthetic data are mixed, as long as the synthetic fraction is not zero. Source
- Alaa and colleagues propose alpha-precision, beta-recall and authenticity as sample-level metrics of fidelity, diversity and generalization, with outliers set aside when judging fidelity and diversity. Source
- The SDMetrics quality report computes Column Shapes and Column Pair Trends scores as averages across columns and column pairs. Source
- Stadler and colleagues found synthetic data either did not prevent inference attacks or did not retain utility, and that outlier records largely remained vulnerable. Source
- The NIST handbook notes that for small samples, normal-approximation confidence limits for a proportion may not be accurate enough, and gives exact binomial limits. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.