Skip to content

AI uses for records

Real vs. synthetic exception benchmark: size the rare-case test first

By SourceX Editorial · Updated

9 sections

Short answer

A real-versus-synthetic exception benchmark trains two arms at matched compute and matched validation budgets, then scores both only on real exception cases held out family by family. The rarest family sets its size: with 50 cases, one arm's pass rate carries about 14 points of uncertainty either way, so only large gaps count. Register families, budgets and verdict rules first. This page is a method; no results are published.

Evaluation split

Real

This page covers real vs. synthetic exception benchmark: size the rare-case test first.Illustrative

Key takeaways

  • Score exceptions family by family on real held-out cases; an average across families can hide the one family where generated data fails.
  • Match two budgets, not one: the same training compute for both arms and the same expert hours for validating synthetic cases as for labeling real ones.
  • With 50 real cases in a family, a 50 percent pass rate carries roughly 14 points of 95 percent uncertainty either way; 100 cases bring that to about 10.
  • Zero failures in 30 cases still leaves room for a true failure rate near 10 percent, so small families support descriptions, not verdicts.
  • Register families, budgets, metrics and the inconclusive rule before training, and publish negative and inconclusive families with the rest.

What does a real-versus-synthetic exception benchmark decide?

It decides, for one task and one generator, whether synthetic exception cases can stand in for real ones without losing performance on the rare situations a model will actually meet. The answer comes family by family: generated cases can be adequate for a manager-approved refund above policy and inadequate for a conflict between two policies in the same dataset.

The benchmark compares two training arms built under identical constraints. One adds real exception cases, the other adds synthetic cases generated to cover the same families, and both are scored only on real exceptions that neither arm, nor the generator, ever saw. Everything else, from the base model to the number of training steps, is held fixed so the data is the only difference.

This page publishes the design. No study has been run under it and no results are reported, so read it as a protocol a buyer, a supplier or a lab can adopt, not as evidence that either arm wins.

Why does an average score hide the exception result?

Rare families contribute few cases to an overall score, so a model can improve on average while getting worse on the cases that matter most. Sagawa and colleagues (arXiv 1911.08731, 2019) found that stronger regularization raised worst-group accuracy by 10 to 40 percentage points on three tasks while average accuracy stayed high, so a healthy average said little about the weakest group.

Synthetic data adds its own reason to look at the tails. Shumailov and colleagues reported in Nature in 2024 that models trained on model-generated content across generations forget the true underlying distribution, with the rare end disappearing first. NIST's Generative AI Profile (NIST AI 600-1) likewise warns that overly homogenized content can contribute to model collapse.

Generated data can also win, and the design has to allow it. The phi-1 paper (arXiv 2306.11644, 2023) reported 50.6 percent pass@1 on HumanEval for a 1.3-billion-parameter code model trained partly on GPT-3.5-generated textbooks and exercises. Because either outcome is possible, the benchmark reports each family separately and treats the worst family as a primary result rather than a footnote.

Matching compute and validation budgets between the two arms

Hold compute fixed by training both arms from the same base checkpoint with the same optimizer settings, number of steps, tokens seen and seeds; hold validation fixed by giving each arm the same number of expert hours. Without the second match, a synthetic arm that received hours of expert filtering ends up compared with a real arm that received none, or the reverse.

Count validation hours where they are actually spent. On the synthetic side that means reviewing seed cases, checking each generated exception against the rule that defines its family and discarding the failures. On the real side it means confirming labels, checking that each case truly belongs to its family and removing records that rights or privacy review excluded. If one arm needs more hours to reach the same number of usable cases, record that as a finding instead of quietly equalizing case counts.

Run several trials per arm. Anthropic's engineering guide to agent evaluations explains that model outputs vary between runs, so it runs multiple trials to get more consistent results; training runs vary too, and a single seed cannot separate a data effect from run-to-run noise. Report the spread across seeds next to every family result.

How many real exception cases does each family need?

Enough that the smallest gap worth acting on is larger than the uncertainty around it, which for most buyers means at least 100 real held-out cases in every family that is to receive a verdict. Smaller families still belong in the report, but as counts with intervals rather than as wins or losses.

The figures use the normal approximation for a binomial proportion and, for the paired column, the standard matched-pairs sample-size formula with an assumed 20 percent of cases on which the two arms disagree, at 95 percent confidence and 80 percent power. The zero-failure column is the exact one-sided 95 percent upper bound, one minus 0.05 raised to the power one over n. Use Wilson or exact intervals in the final report; NIST's e-Handbook of Statistical Methods notes that the Wilson method is recommended by Brown, Cai and DasGupta and by Agresti and Coull.

Thin test sets are a known weakness well beyond this use case. Card and colleagues (EMNLP 2020) found underpowered experiments common in NLP research and estimated that typical machine translation test sets of 2,000 sentences have about 75 percent power to detect a 1 BLEU point difference.

Each row shows the uncertainty a family of that size carries and the verdict it can support.

How many real exception cases does each family need?
Real held-out cases in the family95% margin on one arm's pass rate (at 50% / 80%)Smallest paired gap detectableUpper bound if zero failures are seenVerdict this size supports
30±18 / ±14 pointsNone within reach at 20% disagreement9.5%Report counts only
50±14 / ±11 pointsAbout 17 points5.8%Call only very large gaps
100±10 / ±8 pointsAbout 12 points3.0%Minimum for a family verdict
200±7 / ±5.5 pointsAbout 9 points1.5%Routine family verdicts
400±5 / ±4 pointsAbout 6 points0.7%Fine-grained comparison
Five horizontal interval bars centered on a 50 percent pass rate, narrowing from plus or minus 18 points at 30 real cases to plus or minus 5 at 400, with the 100-case row marked as the minimum for a family verdict.
95 percent margins on one arm's pass rate by family size, normal approximation. Arithmetic, not study results.

Exception families and the fidelity metrics to report

An exception family is a group of cases that share a trigger and the rule that governs the response, written precisely enough that a reviewer can decide membership without knowing which arm produced a case. Illustrative families from operational records include a refund approved above the policy limit by a manager, a three-way match failure resolved by a supplier credit and a field visit rescheduled after the technician could not get access.

Fidelity is measured on the model's behavior on real cases, not on how realistic the synthetic cases look. For each family and arm, the report should give four numbers: the pass rate against the family's rule, the critical-error rate (actions that would breach policy), the worst-family pass rate across all families, and the pass rate on common, non-exception cases, which shows whether either arm bought exception gains with routine losses.

Anchor the held-out set in real work. OpenAI's GDPval (2025) measures models on real-world, economically valuable tasks across 44 occupations; an exception benchmark should likewise draw its held-out cases from actual records with known outcomes, sealed before any generation starts so the generator cannot learn them.

The registration card: what to fix before the first training run

Fix every choice that could be tuned after seeing results, and time-stamp it before training begins. The Center for Open Science describes preregistration as separating hypothesis-generating (exploratory) research from hypothesis-testing (confirmatory) research, so anything decided after the first result belongs in an exploratory section of the report.

The card has seven fields, and each needs an entry before any arm is trained.

The registration card: what to fix before the first training run
FieldWhat to registerIllustrative entry
Task and modelBase checkpoint, prompt format and evaluation harness versionOne pinned checkpoint shared by both arms
Exception familiesName, trigger, governing rule and membership test for each familySix families, each defined by a one-sentence rule
Held-out setSource records, sealing date, custodian and case count per familyAt least 100 real cases in each family that gets a verdict
Compute matchSteps, tokens seen, optimizer settings and seeds for each armIdentical for both arms, three seeds each
Validation matchExpert hours per arm and what counts toward themEqual hours, logged by task type
MetricsFamily pass rate, critical-error rate, worst-family pass rate, common-case pass rateAll four, for every family and arm
Verdict rulesEquivalence margin, the rule for better or worse, and when to declare inconclusiveEquivalent if the paired interval sits within plus or minus 10 points

Which verdicts can the benchmark reach?

Four verdicts are possible for each family, and inconclusive is a legitimate outcome rather than a failed study. Writing them down before training lets a buyer act on the result without relitigating it afterward.

At a glance

Which verdicts can the benchmark reach

  • Synthetic adequate: the paired interval for the family sits inside the registered equivalence margin and critical errors are no higher. The buyer can rely on generated cases for that family and spend its real-data budget elsewhere.
  • Real needed: the synthetic arm falls short beyond the margin on pass rate or on critical errors. The family goes into a sourcing request for real cases.
  • Synthetic better: the synthetic arm leads beyond the margin. Check for leakage between the generator's inputs and the held-out set before accepting the result, then record it.
  • Inconclusive: the interval crosses the margin, usually because the family is too small. Report the counts, keep the family out of purchase decisions and size the next round from the case-count table.

What this design cannot show, and why no results are published

It cannot generalize beyond the task, generator, base model and families it registers; a verdict for one generator version says nothing about the next. It also shows where a synthetic arm falls short but not why, so it pairs naturally with an exception gap report that counts which families the generated data under-represents.

It does not settle cost either. Matching validation hours equalizes effort, not price, and a synthetic arm that ties on every family may still be the cheaper or the dearer option once generation, review and licensing are counted.

SourceX has not run a study under this protocol, and none of the figures on this page are results: the case-count table is arithmetic, the registration entries are illustrative and the verdict rules are proposals. Any study run under it should publish its negative and inconclusive families alongside the rest.

Sourcing real exception cases for the held-out set

The held-out set has to come from real records with known outcomes, and the right to use them for evaluation has to be settled before they are sealed. Datasets for AI buyers come from U.S. businesses on request: SourceX does not hold them in stock, and a buyer describes the scope, volume, language, format, timing and licensing needs it has through the request form.

Exception families are often small, which suits a targeted request. As a general guide, buyers look for years of consistent history and tens of thousands of records or more in one system, but smaller sets can qualify when the work is rare, and each is assessed deal by deal. A request that names each family, the count it needs from the case-count table and the acceptance test lets a supplier say early whether the cases exist; nothing here claims current stock or predicts how a study would turn out.

Frequently asked questions

What is a real-versus-synthetic exception benchmark?

It is a controlled comparison of two training arms, one given real exception cases and one given synthetic cases for the same families, scored only on real held-out exceptions. Compute and expert validation hours are matched so the data is the only difference. Results are reported family by family, including inconclusive ones.

How many test cases does each exception family need?

For a family-level verdict, plan on at least 100 real held-out cases. With that many, a 50 percent pass rate carries about 10 points of 95 percent uncertainty either way, and a paired gap of about 12 points becomes detectable at 80 percent power if the arms disagree on 20 percent of cases. With 30 cases, report counts only.

Is synthetic data better than real data for rare exceptions?

There is no general answer, which is why the benchmark exists. Published work shows both risks and gains: training on generated content can erode the rare end of a distribution, while phi-1 reached 50.6 percent pass@1 on HumanEval with partly synthetic training data. The answer has to be measured for each task and each family.

What does a matched validation budget mean?

Both arms receive the same number of expert hours. Synthetic hours go to reviewing seeds and checking generated cases against each family's rule; real hours go to confirming labels and family membership. Unequal effort would credit whichever arm received more review rather than the data itself.

Has SourceX published results from this benchmark?

No. This page is a protocol; no study has been run under it and no results are reported. Buyers who need real exception cases for a held-out set can describe the families and counts they need through the request form.

Sources

  • Sagawa and colleagues found that stronger regularization raised worst-group accuracy by 10 to 40 percentage points on three tasks while average accuracy stayed high. Source
  • Shumailov and colleagues reported in Nature (2024) that training on model-generated content makes models forget the true underlying distribution, with the tails lost first. Source
  • NIST AI 600-1 notes that overly homogenized content can contribute to model collapse, which can amplify homogenization from the model that generated the training data. Source
  • The phi-1 paper reports 50.6 percent pass@1 on HumanEval for a 1.3-billion-parameter model trained partly on GPT-3.5-generated textbooks and exercises. Source
  • Anthropic's guide to agent evaluations explains that model outputs vary between runs, so multiple trials are run to produce more consistent results. Source
  • The NIST/SEMATECH e-Handbook notes that the Wilson method for proportion confidence intervals is recommended by Brown, Cai and DasGupta and by Agresti and Coull. Source
  • Card and colleagues (EMNLP 2020) found underpowered experiments common in NLP and estimated that 2,000-sentence translation test sets have about 75 percent power to detect a 1 BLEU point difference. Source
  • OpenAI's GDPval measures model performance on real-world, economically valuable tasks across 44 occupations. Source
  • The Center for Open Science describes preregistration as separating hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies