Definitions and comparisons
Synthetic data diversity versus novelty: are different words different decisions?
By SourceX Editorial · Updated
10 sections
Short answer
Different wording is not a different decision. To tell them apart, give every generated task a decision signature: the rule that applies, the facts that change the answer, the correct action and the outcome. Count distinct signatures instead of distinct sentences, then compare them with an untouched holdout of real cases to find paraphrase clusters, invalid inventions and the real exceptions the synthetic set never reaches.
Checklist
Synthetic data diversity
Key takeaways
- Paraphrase filters and embedding spread measure wording; they cannot show that a task asks for a new judgment.
- A decision signature, made of rule, deciding facts, correct action and outcome, is the unit to count.
- Real-only signatures in an untouched holdout reveal the exceptions a synthetic set never reaches.
- Generated-only signatures need a domain validator, because some are rare real cases and some break the policy.
- When wording variety is the goal, report phrasings per signature and signature coverage side by side.
How do you tell wording diversity from diversity of business decisions?
Strip each generated task down to the decision it requires: which rule applies, which facts in the case change the answer, what action or state change is correct, and what outcome follows. Call that its decision signature. Two tasks with different words and the same signature are the same decision. Count distinct signatures, not distinct sentences, and compare them with the signatures in an untouched set of real cases.
Most synthetic data checks measure the text. Paraphrase filters reject near-duplicate wording, embedding metrics reward spread in meaning space, and n-gram counts reward varied vocabulary. All of them can score a set highly when it contains a handful of decisions dressed in thousands of phrasings. For a model that has to act on business rules, such as approving a refund, routing a claim or applying a discount, the decisions are what it learns from.
This page gives a decision novelty audit you can run on any generated task set, a worked example, the case where wording diversity really is the goal, and what to disclose when you ship or buy a synthetic or mixed dataset.
Why paraphrase filters and embedding spread overstate novelty
Paraphrase filters were built to stop copy loops, not to prove that a task is new. Self-Instruct, a widely reused recipe for generating instruction data, adds a generated instruction to its pool only when its ROUGE-L similarity to every existing instruction is below 0.7; the released code computes ROUGE-L against seed and machine instructions and skips any candidate whose maximum score exceeds 0.7. That threshold keeps the pool from repeating itself word for word. It says nothing about whether two instructions that share few words ask for the same judgment.
Embedding-based diversity has the same blind spot at a higher level. A customer asking for a refund on a late delivery and a different customer asking, angrily and at length, for their money back on a parcel that arrived after the promised date sit far apart in wording and may sit apart in embedding space, yet they invoke the same rule and need the same action.
The opposite error also happens. Two tasks that read almost identically can need different actions because one fact differs: the order is outside the return window, the customer is on a contract tier with different terms, or the item is excluded by policy. A paraphrase filter would discard one as a near duplicate and remove a real decision boundary from the set.
Fidelity and coverage metrics for synthetic data do not close the gap either. Alaa and co-authors' sample-level metrics, alpha-precision, beta-recall and authenticity, measure whether synthetic samples look like typical real samples, how much of the real distribution they cover and whether they are copies. They work on whatever representation you give them; if that representation is text embeddings, coverage is coverage of wording.
What counts as one business decision?
A decision is defined by its consequences, not its phrasing: the set of facts that determine the correct action, and the action itself. The clearest way to see this is in agent benchmarks that grade by results. In tau-bench, a benchmark of tool-using agents working under domain policies, an episode's reward depends on whether the database ends in the same state that the task's ground-truth actions produce; the wording of the conversation does not enter that check.
A practical decision signature has four parts. The rule or policy clause that governs the case, with its effective date. The decision-relevant facts, reduced to the values that change the answer, such as inside or outside a window, above or below an approval limit, eligible or excluded. The required action or end state, such as refund issued, partial credit, escalate to a supervisor or decline with a reason. And the outcome category, if the task includes what happened next.
Write the signature schema before generating anything, with a domain validator who knows the real process. If the validator cannot say which facts change the answer, the generator cannot be expected to vary them on purpose.
Keep the deciding facts coarse. A refund window becomes inside or outside, not the exact number of days; an order value becomes below or above the approval limit, not the amount. Coarse values make signatures comparable across real and generated cases, and they stop the count from inflating every time a generator picks a new number that changes nothing about the answer. Where a fact really is continuous and the answer changes along it, record the threshold the policy uses and treat each side of it as a separate value.
The decision novelty audit for generated tasks
Run these steps on a generated task set, in order. Each step leaves an artifact that goes into the dataset's documentation.
| Step | What you compute | Evidence to keep | Red flag |
|---|---|---|---|
| 1. Freeze a real holdout | A sample of real cases set aside before generation and never shown to the generator | Holdout IDs, date range, sampling rule, access log | Holdout cases appear in prompts, seeds or few-shot examples |
| 2. Record the generation recipe | Model, prompts, seeds, grounding sources, filters and the policy version used | Recipe file with hashes and dates | Policy version or grounding sources unknown |
| 3. Assign decision signatures | Rule, decision-relevant facts, required action and outcome for every task, real and generated | Signature table and the labeling guide used | Many tasks the validator cannot assign a signature to |
| 4. Collapse paraphrases | Group generated tasks by identical signature and count tasks per group | Group sizes and a few examples per group | A few groups hold most of the tasks |
| 5. Compare coverage with the holdout | Signatures shared, real-only and generated-only | Three lists with counts | Many real-only signatures, especially rare ones |
| 6. Validate generated-only signatures | Whether each is a rare but possible case or an impossible one under the dated policy | Validator decision and reason for each | Generated-only signatures that break the policy |
| 7. Check for copies and privacy leakage | Whether generated rows match or sit unusually close to real training rows | Exact-match and distance-to-closest-record results against a holdout | Generated rows closer to training rows than holdout rows are |
| 8. Run negative controls | Planted tasks with a known wrong answer or a deliberately changed fact | Which controls the signature labeling caught | Labeling misses facts that should flip the decision |
Which real decisions never show up among the generated tasks?
Real-only signatures, the decisions present in the holdout and absent from the generated set, are usually the most important output of the audit. They are the exceptions a model trained on the synthetic set has never seen.
There is a known reason to expect them. Shumailov and co-authors, writing in Nature, report that training models indiscriminately on model-generated content causes irreversible defects in which the tails of the original content distribution disappear. A generator asked for more refund tasks tends to produce the common refund cases, because those dominate what it learned.
Rank real-only signatures by how often they occur in the holdout and by their business cost. A rare signature that triggers a large write-off or a regulatory step deserves more attention than a common one with a trivial action. For each, decide whether to generate it deliberately from a grounded template, source more real cases, or accept the gap and document it.
Separating novel decisions from invalid ones
A generated-only signature is either a real but rare case that the holdout happened to miss, or a case the policy does not allow. Only a domain validator working from the dated policy can tell which.
Common invalid patterns: a discount stacked with an offer the policy says cannot be combined, an approval granted above the agent's limit without escalation, or a fact pattern that cannot occur in the real system, such as a refund on an order that was never shipped and never charged. Remove these or relabel them as negative examples, never as correct behavior.
Record the policy date for every signature. A rule that changed in the middle of the generation window splits one signature into two, and a model should not learn the old rule as if it were current.
Worked example (illustrative): two thousand generated refund tasks
The figures below are invented to show the audit's outputs. A team generated tasks to train a support agent on a retailer's refund policy and set aside an untouched holdout of real refund cases.
- Decision, based on the audit outputs in the table below: cap any one signature group, generate targeted tasks for the most costly real-only signatures from grounded templates, and request more real cases for the rest.
- Why the paraphrase filter missed it: the 2,000 tasks were textually distinct, so every text check passed. Only the signature count showed that one decision filled more than a third of the set while 22 real exceptions were absent.
- What the example does not show: whether the agent trained on the revised set performs better. That needs a separate evaluation on the holdout.
| Audit output | Illustrative count | What it means |
|---|---|---|
| Generated tasks after the paraphrase filter | 2,000 | Text-level checks pass; wording is varied |
| Distinct decision signatures among generated tasks | 23 | Most tasks are rephrasings of a few decisions |
| Largest signature group | 38 percent of generated tasks | One case, a late delivery inside the window, dominates |
| Distinct signatures in the real holdout of 400 cases | 41 | The real process has more decisions than the generated set |
| Shared signatures | 19 | These are covered by both |
| Generated-only signatures | 4, of which 2 broke the policy | Two rare valid cases kept; two invalid tasks relabeled as negatives |
| Real-only signatures | 22 | Exceptions such as partial shipments and excluded items never appeared |
When different wording is the thing you need
Sometimes surface variety is the goal, and the audit should say so rather than penalize it. Intent classifiers, speech and text robustness tests, and retrieval systems that must match many phrasings to one answer all benefit from many ways of saying the same thing.
In those cases keep the decision signature anyway and report two numbers: phrasings per signature, which you want high, and signatures covered against the holdout, which still tells you whether the set reaches the cases that matter. A set with many phrasings of the wrong decisions is still the wrong set.
The audit also has limits. Signatures depend on the validator's schema, and a schema that misses a decision-relevant fact will merge cases that should be separate. Review the schema against the holdout's hardest cases before trusting the counts.
What a seller or buyer of a synthetic mixture should disclose
- The generation recipe: model, prompts, seeds, filters and grounding sources, with dates.
- The policy or rule version each task was generated against.
- The signature schema, the signature coverage table against a named real holdout, and a statement that the holdout was never shown to the generator.
- Group sizes for the largest signature groups and any caps applied.
- Generated-only signatures, the validator's decision on each, and how invalid ones were handled.
- Copy and privacy screen results, such as exact-match and distance-to-closest-record checks against real rows. SDMetrics' NewRowSynthesis measures whether each synthetic row is new or exactly matches a real row, and its DCR overfitting protection metric compares distances to the closest training record against a holdout.
- The share of real and generated tasks in any mixture, and which split each was used in.
Where real records fit in a decision novelty audit
The audit cannot run without real cases: they define the holdout, reveal the missing exceptions and anchor the policy dates. Teams that have only generated data can count phrasings, but they cannot tell which decisions are missing.
SourceX does not keep stock datasets; each request is matched to U.S. businesses whose records suit it. For a novelty audit, that request can name the workflow, the decision types missing from the generated set or needed for the holdout, and the volume, language, format, timing and licensing terms.
Frequently asked questions
What is the difference between diversity and novelty in synthetic data?
Diversity usually means variation in the data's surface, such as wording, vocabulary or embedding spread. Novelty in the sense used here means a task requires a decision that other tasks do not: a different rule, a different set of deciding facts or a different correct action. A set can be highly diverse and add almost no new decisions.
Does a ROUGE-L or embedding similarity filter make generated tasks unique?
It makes them textually distinct. Self-Instruct's filter, for example, rejects instructions with ROUGE-L similarity above 0.7 to existing ones, which prevents copying but does not check whether two dissimilar texts require the same judgment. It can also discard near-identical tasks that differ in one deciding fact.
How large should the real holdout be for the audit?
Large enough to contain the decision types you care about, including the rare ones, which depends on the process rather than a fixed number. Sample it across the full date range and case types, freeze it before generation and keep it away from the generator's prompts and seeds.
Who should define the decision signatures?
A domain validator who knows the real process, working with the team building the model. They name the rules, the facts that change the answer and the correct actions, and they review generated-only signatures against the dated policy.
Can the audit be automated?
Parts of it can: grouping by signature, computing coverage and running copy checks. Assigning signatures to free-text tasks can be assisted by a model, but a validator should check a sample and every generated-only signature, because those are where errors cost the most.
Sources
- The Self-Instruct code computes ROUGE-L similarity between a candidate instruction and existing seed and machine instructions and skips candidates whose maximum similarity exceeds 0.7. Source
- The Self-Instruct paper adds a generated instruction to the task pool only when its ROUGE-L similarity with existing instructions is below 0.7, to encourage diversity. Source
- tau-bench sets an episode's reward to zero unless the resulting database state matches the state produced by the task's ground-truth actions. Source
- SDMetrics' NewRowSynthesis metric measures whether each synthetic row is new or exactly matches a row in the real data. Source
- SDMetrics' DCROverfittingProtection metric uses distance to closest record to measure whether synthetic data is overfit to the real data, compared with a holdout set. Source
- Shumailov and co-authors report in Nature that indiscriminate training on model-generated content causes irreversible defects in which the tails of the original content distribution disappear. Source
- Alaa and co-authors propose alpha-precision, beta-recall and authenticity as sample-level metrics for fidelity, coverage and copying in synthetic data. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.