Skip to content

AI uses for records

Synthetic data exception preservation: how to find what it dropped

By SourceX Editorial · Updated

10 sections

Short answer

To detect business exceptions missing from synthetic training data, write each exception as an executable rule, count it in the synthetic set and in an untouched real holdout, compare the rates with exact intervals, and check that every synthetic exception obeys the rule that defines it. Classes that are absent, rarer or internally inconsistent go into a gap report, alongside how the synthetic data was grounded, generated and selected.

Agent run

Synthetic data exception

This page covers synthetic data exception preservation: how to find what it dropped.Illustrative

Key takeaways

  • Aggregate fidelity scores can look good while rare business exceptions are missing, because they average over columns or set outliers aside.
  • An exception must be defined as a rule over fields, such as a refund above the policy limit with an approver recorded, before it can be counted.
  • Compare exception rates against a real holdout that never touched the generator, using exact binomial intervals for rare classes.
  • Count is not enough: every synthetic exception should also satisfy the rule that defines it, or it teaches the wrong behavior.
  • A class too rare in the real holdout to measure is a finding in itself; it calls for more real records, not a pass.
  • Ask synthetic suppliers to disclose grounding records, generation recipe, selection filters and the policy date their rules reflect.

How do you detect business exceptions missing from synthetic training data?

Define each exception as a rule over fields, count it in both the synthetic set and an untouched real holdout, compare the rates with intervals, and test whether each synthetic exception is internally consistent. Anything absent, under-represented or inconsistent is a gap, and the gap report is what you accept or reject the data on.

Business exceptions are the cases where normal processing stops: a refund above the policy limit, an invoice that fails matching, a shipment split across warehouses, a ticket reopened after closure. They are often the cases a model or agent is bought to handle, and they are exactly where generated data tends to be thin.

The method works on the data itself, before any model is trained. That makes it cheap to run on a vendor sample, and it gives a buyer specific, checkable reasons to accept, renegotiate or reject a synthetic or mixed dataset.

Why does synthetic data smooth away exceptions?

Generators learn the bulk of a distribution more easily than its tails, and common quality metrics are built to summarize the bulk. Both push in the same direction: rare combinations of fields fade, and the scorecard does not show it.

Research on recursive training shows the mechanism at its extreme. Shumailov and colleagues, writing in Nature, found that training on model-generated content causes defects in which tails of the original content distribution disappear. A later theoretical study reports that this collapse can persist even when real and synthetic data are mixed, as long as the synthetic fraction is not zero. Your vendor's generator may not be recursive, but the lesson about tails carries over.

Standard metrics can hide the loss. The alpha-precision and beta-recall metrics proposed by Alaa and colleagues measure fidelity and diversity at sample level, and by design they set outliers aside when doing so. The SDMetrics quality report scores Column Shapes and Column Pair Trends as averages across columns and pairs. A refund-override pattern that lives in three fields and one percent of rows barely moves either kind of score.

What counts as a business exception in operational records?

A business exception is a record, or sequence of records, where a stated rule was triggered and something other than the default happened. To be testable it needs a predicate: fields and values that decide, without judgment, whether a row is in the class. The table gives illustrative classes from common systems; replace the thresholds with your own policy values.

What counts as a business exception in operational records?
Exception class (illustrative)Typical systemRule that defines itConsistency check
Refund above policy limitSupport desk or order systemrefund_amount greater than policy_limitapprover_id present and approval timestamp before refund timestamp
Invoice failed three-way matchERP accounts payableinvoice quantity or price differs from PO or receipt beyond tolerancehold status set, then a release or credit memo follows
Partial shipment with backorderOrder management or WMSshipped_qty less than ordered_qty on a linebackorder line exists and later ship date is after first
SLA breach with escalationTicketing systemfirst response time greater than SLA targetescalation event exists and follows the breach
Ticket reopened after closureTicketing systemstatus changes from closed to opennew customer message or agent note between close and reopen
Duplicate customer mergedCRMmerge event links two customer idssurviving id keeps history of both; merged id stops receiving activity
Chargeback after refundPayments and order systemchargeback event on an order already refundeddispute outcome recorded; no second refund issued

How do you run the exception frequency comparison?

Run seven steps in order, and freeze the real holdout before anything else, because a holdout that informed the generator cannot reveal what the generator missed. The steps take a data engineer and a process owner, not a research team.

Use exact intervals for rare classes. NIST's engineering statistics handbook notes that when the sample is small, confidence limits approximated with the normal distribution may not be accurate enough, and it gives exact binomial limits instead. Exception classes are small samples by definition, so compute exact (Clopper-Pearson) intervals for each class rate in both sets.

  • 1. Freeze a real holdout from the same systems and period, never used for grounding, prompting, fine-tuning or filtering the generator. Record its date range and policy version.
  • 2. Write each exception class as a predicate over fields, reviewed by the process owner who knows the policy. Version the predicates.
  • 3. Count each class in the holdout and in the synthetic set, as rows and, for sequence classes, as cases (ticket, order, invoice).
  • 4. Compute each class rate with an exact binomial interval in both sets.
  • 5. Run the consistency check on every synthetic row in each class: required fields present, events in a valid order, values within system limits.
  • 6. Check co-occurrence for pairs of classes that occur together in the holdout, such as an SLA breach followed by a refund override.
  • 7. Write the gap report and attach the generation disclosure from the supplier.

What should the exception gap report include?

The report needs one row per exception class with both rates, their intervals, a consistency pass rate and a verdict, plus a header that states the holdout's date range, the predicate version and the generator disclosure. A reviewer should be able to reproduce every number from the predicates and the two datasets.

Use the fields and flag rules below as is. Thresholds are starting points for a technical reviewer, not standards; set them before you look at the results so the verdicts cannot drift toward acceptance.

What should the exception gap report include?
Report fieldDefinitionHow to computeFlag rule
Real rate and intervalShare of holdout rows or cases in the classCount over total, exact binomial 95% intervalInconclusive if fewer than about 10 real occurrences
Synthetic rate and intervalShare of synthetic rows or cases in the classSame method on the synthetic setMissing if zero synthetic occurrences
Rate ratioSynthetic rate divided by real ratePoint ratio; note whether the two intervals overlapUnder-represented if intervals do not overlap and synthetic is lower
Consistency pass rateShare of synthetic class rows passing the consistency checkPassing rows over class rowsDistorted if below the threshold set in advance
Co-occurrence gapClass pairs common in the holdout but absent together in synthetic dataPair counts in both setsFlag any pair present in the holdout and absent in synthetic data
VerdictPreserved, under-represented, missing, distorted or inconclusiveApply the flag rules in orderAny missing or distorted class blocks acceptance for that use

Worked example: refunds and escalations in support tickets (illustrative)

Every number here is invented to show the method; it is not a study result. A buyer evaluates a vendor's synthetic support-ticket set of 100,000 tickets, generated from a seed the vendor describes, against a frozen real holdout of 20,000 tickets from one company's help desk covering two years.

In the holdout, refunds above the policy limit appear in 180 tickets; in the synthetic set, in 310. The rates are about 0.9% and 0.3%, the exact intervals do not overlap, and the class is under-represented. Worse, only 40% of the synthetic overrides carry an approver, against every one in the holdout, so the class is also distorted: a model trained on it would learn that large refunds need no approval.

Tickets reopened after closure appear in 1,150 holdout tickets and at a similar rate in the synthetic set, with a high consistency pass rate; that class is preserved. Chargebacks after a refund appear 46 times in the holdout and never in the synthetic set: missing.

The exception that matters most is the one the method cannot decide. Refunds issued in a currency other than the account currency appear only four times in the real holdout. The interval is too wide to judge any synthetic rate against, so the verdict is inconclusive, and the right action is to source more real records of that class, not to accept the synthetic version on trust.

What should a synthetic data supplier disclose?

Ask for the derivation: which real records grounded the generator, how examples were generated and filtered, and which policy date the rules reflect. Without that, a gap can be a generator limit, a selection filter or an outdated rule, and each needs a different fix.

At a glance

What should a synthetic data supplier disclose

  • Real holdout: confirmation that the buyer's or a held-back real set never touched generation, with its date range
  • Generation recipe: model or method, prompts or conditioning, and any class balancing applied
  • Grounding sources: which real records seeded the generator and under what rights
  • Selection filters: what was dropped after generation, such as outliers, failed validations or near-duplicates
  • Policy date: the version of business rules the synthetic records follow, and whether it changes over the series
  • Rule consistency: the supplier's own constraint checks and their pass rates
  • Privacy screen: how memorized or near-copied real records were detected and removed
  • Negative controls: deliberately invalid examples, if any, and how they are labeled

Which exception shapes do generators most often drop?

The classes most often missed share a structure: they need several fields, several records or a change over time to exist. Single-field rarities are easier for a generator to reproduce than multi-step ones, so check these first.

Sequence exceptions, such as reopen after close or chargeback after refund, need ordered events across a case. Cross-record exceptions, such as duplicate customers or split shipments, need two or more linked rows. Policy-change exceptions need records before and after a rule changed, which a generator conditioned on one policy cannot produce. Timing exceptions, such as a dispute filed months after an order, need long gaps that short generation windows rarely include.

To test these classes, extend the predicates beyond single rows. A sequence predicate checks event order within a case, a cross-record predicate joins on the shared key, and a policy-change predicate splits the holdout at the rule's effective date and compares each side separately. If the synthetic set has no case ids, timestamps or join keys, these classes cannot be tested at all, and that absence belongs in the gap report as a finding rather than being left out.

Mixtures need the same check per source. When a supplier blends real and generated records, ask for a flag on each row showing its origin, and run the comparison on the generated rows alone as well as on the blend. A blend can look preserved overall while every generated row in a class is inconsistent.

Which gap-report results should block, narrow or escalate acceptance?

Stop accepting the set for a use when any class that use depends on is missing or distorted; narrow the use when gaps sit in classes it does not need; request specialist review when results are inconclusive or when real exceptions are also privacy outliers.

That last case is common. Stadler and colleagues found that synthetic data either did not prevent inference attacks or did not retain utility, and that outlier records largely remained vulnerable. Rare exceptions are often outliers, so a synthetic set that preserves them faithfully may also expose the real cases behind them. A privacy reviewer should check those classes specifically.

Bring in a domain validator when predicates are disputed. If the process owner and the supplier disagree about what counts as an override or a valid escalation, settle the rule in writing before rerunning the comparison.

Where do real records for the missing exception classes come from?

When the gap report shows missing or inconclusive classes, the fix is usually more real records of those classes. Because SourceX sources datasets for AI teams from U.S. businesses on request, with no standing inventory, a request can list the exception classes and predicates straight from your gap report.

On the supply side, buyers usually want two or more years of consistent history and tens of thousands of records from one system, but SourceX's general guide leaves room for smaller sets when the work is rare, which is where exception-heavy slices often sit; each is assessed deal by deal.

Frequently asked questions

Is synthetic data better than real data for rare cases?

Not by default. Generators tend to reproduce common patterns more faithfully than rare ones, and averaged quality scores can hide that. Synthetic data can help when it is checked class by class against real records and its gaps are filled with real examples.

What are the disadvantages of using synthetic data for business processes?

The main ones are missing or distorted exceptions, rules that reflect one policy date only, and inconsistent multi-step sequences. Rare real cases can also be privacy outliers, so preserving them faithfully can carry re-identification risk.

How large does the real holdout need to be?

Large enough that each exception class you care about appears often enough to give a usable interval; around ten or more occurrences per class is a practical floor. If a class is rarer than that, the holdout cannot judge it and you need more real records of that class.

Can I use the same real data to ground the generator and to test it?

No. A holdout that informed the generator, its prompts or its filters cannot show what the generator missed. Freeze the holdout first and record that it was never used for generation.

Do column-level quality scores catch missing exceptions?

Usually not. Scores such as column shape and column pair similarity are averaged across columns, and some sample-level metrics set outliers aside by design. An exception defined over three fields and one percent of rows barely moves them.

Sources

  • Shumailov and colleagues found that training on model-generated content causes defects in which tails of the original content distribution disappear. Source
  • A theoretical study reports that model collapse can persist when real and synthetic data are mixed, as long as the synthetic fraction is not zero. Source
  • Alaa and colleagues propose alpha-precision, beta-recall and authenticity as sample-level metrics of fidelity, diversity and generalization, with outliers set aside when judging fidelity and diversity. Source
  • The SDMetrics quality report computes Column Shapes and Column Pair Trends scores as averages across columns and column pairs. Source
  • Stadler and colleagues found synthetic data either did not prevent inference attacks or did not retain utility, and that outlier records largely remained vulnerable. Source
  • The NIST handbook notes that for small samples, normal-approximation confidence limits for a proportion may not be accurate enough, and gives exact binomial limits. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies