Skip to content

AI uses for records

Synthetic data distribution mismatch: testing too-polite customers

By SourceX Editorial · Updated

10 sections

Short answer

Assess synthetic data distribution mismatch by labeling synthetic conversations and an untouched real holdout with the same blind rubric, then comparing behavior and exception rates, not just topics: frustration, escalation, out-of-policy requests, missing details, multi-issue contacts and length. Flag any real exception type missing from the synthetic set, and confirm with a model trained on synthetic data and scored on real cases.

Agent run

Synthetic data distribution

This page covers synthetic data distribution mismatch: testing too-polite customers.Illustrative

Key takeaways

  • Topic mix can match while behavior does not; compare frustration, escalation, exceptions and length.
  • Label real and synthetic conversations together with one rubric, blind to which is which.
  • Start from the real exception list and look for each type in the synthetic set.
  • Fidelity scores can reward typical samples, so report coverage of real categories too.
  • Intended mismatches such as oversampling are fine when documented and corrected for in evaluation.

How do you assess distribution mismatch when synthetic customers behave unrealistically?

Compare the synthetic set against an untouched real holdout on behavior and exceptions, not only on topics. Label both sets with the same rubric, blind to which is which, then compare the rate of each behavior: frustration, escalation requests, abandoned threads, requests outside policy, wrong or missing details, several issues in one contact and conversation length. Finish by training on the synthetic mixture and scoring on real cases.

Synthetic data distribution mismatch usually hides in those columns. A generator can match the mix of billing, shipping and account questions closely while producing customers who are calmer, clearer and more cooperative than real ones, and a model trained on them learns a world without the hard cases. The comparison below is built to expose that.

Why generated customers drift toward the calm, complete and typical

Several published findings point in the same direction. Kirk and colleagues report that RLHF fine-tuning substantially decreases the diversity of outputs sampled for a given input compared with supervised fine-tuning, and that RLHF produces less diverse text on some metrics even across inputs. A generator built on an assistant model inherits that narrowing.

Li and colleagues found that subjectivity, at both the task and instance level, is negatively associated with the performance of models trained on LLM-generated data, and they point to limited diversity in generated data on subjective tasks. Customer tone, frustration and whether a request is reasonable are subjective in exactly that sense.

Shumailov and colleagues show in Nature that training on recursively generated data makes the tails of the original distribution disappear. A single round of generation is not recursion, but the lesson carries over for evaluation: the rare, messy cases are the first to thin out, so they are the ones to count.

The behavior and exception distribution comparison

Use one row per behavior the downstream model must handle. Set the flag rule before generating the comparison, and keep the real holdout out of the generator's prompts, examples and retrieval sources.

The behavior and exception distribution comparison
DimensionHow to label itStatisticFlag when
Topic or intent mixExisting taxonomy from the ticketing systemTotal variation complement per category setAny intent's share moves outside the band set in advance
Tone and frustrationThree or four rubric levels, labeled blind to sourceShare per level and total variationTop frustration level under-represented beyond the band
Escalation and supervisor requestsExplicit request to escalate, cancel or complainRate ratio synthetic to real with an intervalInterval excludes the agreed ratio
Requests outside policyRefunds, exceptions or terms the policy does not allowCategory coverage of real exception typesAny real exception type absent from synthetic
Wrong or missing detailsMissing order number, wrong account, contradictory factsRate and coverage of error typesSynthetic customers almost always supply complete details
Several issues in one contactCount of distinct issues per conversationDistribution of countsSingle-issue share materially higher than real
Conversation lengthTurns per conversationSequence length similarityLong threads missing from the synthetic tail
Abandonment and reopenCustomer stops replying, or reopens a closed caseRate per conversationNear-zero synthetic rate where real rate is visible
Off-topic and abusive contentRubric categories after the same safety filterRate before and after filteringFilter removed real anger along with abuse

Building the real holdout and the shared labeling rubric

The comparison is only as good as the holdout. Draw it from the same channels, products and period the model will serve, take it before any generation starts, and never let it reach the generator as seed examples, few-shot prompts or retrieval context. If policies changed during the period, align the holdout and the generation prompts to the same policy date, or an honest exception in the real data will look like a mismatch.

Label both sets with the same rubric and the same labelers, shuffled together so nobody knows which conversation is synthetic. Agree the rubric definitions in writing, especially for frustration and out-of-policy requests, and measure labeler agreement on a shared sample. A rubric that labelers apply inconsistently produces mismatches that belong to the labeling, not the generator.

The real holdout also needs its own privacy screen, since it holds actual customer records. Remove direct identifiers before labeling, and keep the holdout under access controls separate from the synthetic pipeline.

Matching each statistic to the dimension it can measure

Categorical behaviors and continuous ones need different statistics, and averages hide tails. The open-source SDMetrics library documents several single-column measures that fit this comparison: TVComplement is the complement of total variation distance for categories, KSComplement returns one minus the two-sample Kolmogorov-Smirnov D statistic for continuous columns, CategoryCoverage is the fraction of real categories that appear in the synthetic data, and SequenceLengthSimilarity applies the KS comparison to sequence lengths.

Be careful with fidelity scores that reward typical samples. Alaa and colleagues define alpha-precision as the probability that a synthetic sample falls in the alpha-support of the real distribution and beta-recall as the share of real samples covered by the synthetic distribution; their framing deems a sample high fidelity if it looks typical. A set of polite, well-formed customers can score well on precision while recall over the real exceptions is poor, so report both.

For tabular synthetic data, NIST's SDNist report tool evaluates the utility and privacy of a deidentified dataset against benchmark data and is a useful reference for how a full report is laid out, even though conversation data needs the behavior labels above first.

Does the gap change what the model does on real cases?

A distribution gap matters when it changes model behavior on the cases that count, so test that directly. Train one model on real data only, one on the synthetic mixture and, if the plan is to blend, one on the blend at the intended ratio; score all of them on the same real holdout, broken out by the behavior rows in the comparison table rather than as one overall number.

The breakdown is the point. An overall accuracy or win rate can rise with synthetic data because routine intents get easier, while escalation handling, out-of-policy refusals or multi-issue replies get worse. Report each row with an interval, because the rare behaviors have few real examples and their scores move a lot between runs.

Keep the scoring honest in two ways. Use graders or reference answers built from the real holdout only, never from the generator, and freeze the holdout once scoring starts so nobody tunes the generator against it. If the team does tune against it, draw a fresh holdout for the final check.

When the blended model wins overall and loses on one exception row, that row decides the next step: regenerate for it, cap the synthetic share for that intent or add licensed real examples of it. Record which choice was made and why in the mixture's documentation.

Finding the real exceptions the generator never produced

List the exceptions from the real holdout first, then look for them in the synthetic set, rather than the other way around. Starting from the synthetic side only shows what the generator chose to produce.

Negative controls help separate generator gaps from pipeline gaps. Seed a small number of known exception scenarios into the generation prompts and check that they survive every later step; if they vanish, a filter or deduplication rule removed them, not the generator.

Grounding is the other lever. A generator prompted with a generic persona such as an upset customer tends to produce a stereotype, while one prompted with the structure of a real exception, such as the policy clause at issue, the missing detail and the channel switch, produces something closer to the real case. Record which grounding records were used, so a later reviewer can confirm none of them came from the holdout and that no customer details were copied into the synthetic text.

  • Customers who contradict themselves or correct earlier details mid-conversation.
  • Requests the policy does not allow, including repeated requests after a refusal.
  • Threads where the customer goes silent, then returns days later on a different channel.
  • Messages with typos, fragments, pasted screenshots described in text, or several languages.
  • Anger that is not abusive, which safety filters often remove together with abuse.
  • Contacts that bundle a complaint, a question and an account change in one message.

Worked example: an illustrative support-chat mixture

Illustrative only; no real measurements. A team plans to fine-tune a routing and reply model on a mixture of real and generated support chats. It holds out 2,000 real conversations from the latest two quarters before generation, labels them together with 2,000 sampled synthetic conversations, and sets flag bands in advance.

In this illustration the intent mix passes. The behavior rows do not: escalation requests appear in about 9% of real conversations and 2% of synthetic ones, multi-issue contacts in 18% against 4%, and three of the eleven real out-of-policy request types never appear in the synthetic set. The real conversation-length distribution has a long tail of threads above twenty turns that the synthetic set lacks. A model trained on the mixture and scored on the real holdout misroutes escalation requests far more often than one trained on real data alone.

The team regenerates with prompts grounded in the real exception list and drops a tone filter that had removed non-abusive anger, then reruns the comparison against a fresh holdout. The exception: one rare refund type was deliberately oversampled in the synthetic set to give a classifier enough examples. That mismatch is intended, so it is documented as a weighting choice and the evaluation on real data reweights it.

When is a mismatch acceptable, and when should the mixture be held back?

A mismatch is acceptable when it is intended, documented and corrected for in evaluation, such as oversampling a rare case. It is a reason to hold the mixture when it removes behaviors the deployed model must handle, or when the train-on-synthetic, test-on-real result is worse than the real-only baseline on those behaviors.

  • Hold: a real exception category has no synthetic counterpart and the model will meet it in production.
  • Hold: the gap concentrates in one customer segment, language or channel.
  • Narrow: the comparison passes for routine intents but not for disputes; limit the synthetic share to routine intents.
  • Bring in a domain validator: labelers disagree on frustration or policy labels more than on topics.
  • Bring in a privacy reviewer: synthetic conversations reproduce near-verbatim details from real records.

What should a synthetic mixture disclose to the people training on it?

Ship the derivation with the data: the generator and version, the grounding sources and policy date, the prompts or templates, every filter applied, the mixture ratio with real records, and the comparison table with its flag rules and results. Without that record, a buyer cannot tell whether a gap was a choice or an accident.

Re-run the comparison whenever the generator, its grounding records, the mixture ratio or the underlying policy changes, and date each run. A comparison that passed for one generator version says little about the next one.

Real records are what make this comparison possible, both as the holdout and as grounding for better generation. SourceX sources operational datasets from U.S. businesses when a buyer asks for them, instead of keeping an inventory; the request covers scope, volume, language, format, timing and licensing needs, and the supplying company's rights and preparation are reviewed before delivery. A licensed real holdout should be scoped for evaluation use in that agreement if it will never be used for training.

Frequently asked questions

What is synthetic data distribution mismatch?

It is the difference between how often behaviors, values or categories occur in synthetic data and how often they occur in the real data the model will face. Mismatch in rare exceptions matters most, because averages can match while the hard cases are missing.

Is a high fidelity score enough to trust synthetic customer data?

No. Some fidelity measures reward samples that look typical, so a set of calm, complete customers can score well while real exceptions are missing. Report coverage of real categories and test a model trained on synthetic data against real cases.

How big should the real holdout be?

Large enough that the rarest behavior you care about appears often enough to estimate its rate with a usable interval. Count the expected occurrences of that rare behavior first, then size the holdout from it rather than from the total.

Can the same real data seed the generator and test it?

Not for the comparison. Records the generator saw cannot show what it fails to produce, so keep a separate holdout drawn before generation and never used in prompts, examples or retrieval.

Why do synthetic customers come out too polite?

Generators built on assistant models tend to produce less diverse text, and safety or quality filters often remove non-abusive anger along with abuse. The result is customers who are calmer and clearer than real ones, which the behavior comparison is designed to catch.

Sources

  • Kirk et al. report that RLHF substantially decreases the diversity of outputs sampled for a given input compared with SFT, and produces less diverse text on some metrics across inputs. Source
  • Li et al. found subjectivity at the task and instance level is negatively associated with the performance of models trained on LLM-generated data. Source
  • Shumailov et al. show in Nature that training on recursively generated data causes the tails of the original content distribution to disappear. Source
  • SDMetrics defines TVComplement as the complement of the total variation distance between real and synthetic categories. Source
  • SDMetrics KSComplement uses the two-sample Kolmogorov-Smirnov test and returns 1 minus the D statistic. Source
  • SDMetrics CategoryCoverage computes the fraction of real data categories present in the synthetic data. Source
  • SDMetrics SequenceLengthSimilarity compares real and synthetic sequence lengths by applying KSComplement. Source
  • Alaa et al. define alpha-precision, beta-recall and authenticity for synthetic data, with alpha-precision deeming a sample high fidelity if it looks typical. Source
  • NIST's SDNist tool evaluates the utility and privacy of a deidentified dataset and generates a summary quality report. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies