Skip to content

AI data market

Synthetic data policy version fidelity: testing which rule a generator applied

By SourceX Editorial · Updated

9 sections

Short answer

To test whether a generator applied the wrong historical policy version, date every synthetic case, resolve the rule version that governed on that date from a versioned rule table, and score the case's policy-bearing fields against that version and its neighbors. Cases matching a neighbor are version errors. Compare the rates with untouched real cases from the same period and plant deliberate errors to prove the test catches them.

Value drivers

Synthetic data policy

This page covers synthetic data policy version fidelity: testing which rule a generator applied.Illustrative

Key takeaways

  • Statistical fidelity and policy version fidelity are different properties; a synthetic set can match real distributions while applying today's rule to every historical case.
  • Generators drift toward the current rule because prompts, retrieval indexes, static constraints and model knowledge are all newer than the cases they imitate.
  • The test needs a versioned rule table with valid-from and valid-to dates and the anchor event each rule keys on.
  • Cases consistent with both versions prove nothing and should be excluded from the error rate.
  • An untouched real holdout from the same periods separates generator errors from genuine late application of old rules.

How do you test whether a generator applied the wrong historical policy version?

Give every synthetic case an as-of date, resolve which policy version governed on that date, and check the case's policy-bearing fields against that version and against its neighbors. A case that matches a neighbor version but not the governing one is a version error. Then compare the rate of each outcome with untouched real cases from the same period, which tells you whether the generator, not the policy, is the problem.

Policy version fidelity is a different property from the statistical fidelity most synthetic data reports measure. A generated set can match the real data's column distributions, ranges and correlations and still apply today's return window, refund rule or approval threshold to a case dated two years ago. A model trained on that set learns that the current rule always applied, and it will misjudge any historical or transitional case it later sees.

The test below needs four inputs: a dated rule table, a rule evaluator for the fields the policy governs, the synthetic set with its as-of dates, and a real holdout that the generator never touched. Everything else is bookkeeping.

Why generators drift toward the current rule

Because almost every input to a generator is newer than the cases it is asked to imitate. The prompt cites the current policy document, the retrieval index returns the latest version of a help article, and the language model's own knowledge has an effective cutoff that may not match the vendor's stated date. Research on knowledge cutoffs found that effective cutoffs often differ sharply from reported ones and vary by resource, which means a model can be confidently out of date on one policy and current on another.

Tabular generators fail differently but end up in the same place. Libraries such as SDV let teams define business rules as logical constraints, which is useful, but a constraint written once encodes one version of the rule and applies it to every row regardless of date. Unless the constraint takes the as-of date as an input, the generator will enforce the current threshold across the whole history.

Standard quality metrics will not catch this. SDMetrics, for example, compares synthetic data to real data across quality and privacy measures, and its BoundaryAdherence metric checks whether synthetic values respect the minimum and maximum bounds set by the real data. A refund promised under the wrong rule can sit comfortably inside those bounds. Version fidelity needs a rule-aware check keyed on the date.

What the versioned rule table needs before the test can run

It needs every version of each rule with the date it started and stopped governing, and the anchor event each rule keys on. Without the anchor, two analysts can resolve the same case to different versions and both be right by their own reading.

The structure is the familiar type 2 slowly changing dimension. dbt's snapshot feature, for instance, implements type 2 slowly changing dimensions over mutable source tables, adding dbt_valid_from and dbt_valid_to columns so that each version of a row carries the interval it was current. A policy table built the same way lets you join any case to the version valid at its anchor date with one range condition.

Record these fields per version, and fill them from the policy documents themselves, not from the generator's description of them:

  • Rule identifier and version label, stable across systems.
  • Valid from and valid to dates, with the time zone, and whether the end date is inclusive.
  • Anchor event: the case date the rule keys on, such as purchase date, request date, event date or decision date.
  • The policy-bearing fields the rule governs, for example eligibility, amount type, deadline or remedy offered.
  • An executable form of the rule: a function or table that returns the expected value of each governed field for a case.
  • Transition terms: grandfathering, phase-in dates and any period in which both versions were acceptable.
  • Source document and retrieval date, so a reviewer can confirm the version text.

The policy version consistency test

Run the eight steps below on every synthetic set before it is mixed into training or used for evaluation. The output is a per-case label and a summary table by rule, version and period, which is what you report to the dataset's users.

Read the summary by period, not as one overall rate. A wrong-version rate that is near zero for recent months and high for older ones is the signature of a generator anchored on the current rule. A rate that is high only in the weeks around a change may be faithful to a messy transition, which step 7 will confirm or rule out. Unexplained cases that cluster under one rule usually point to an extractor problem or a missing version in the table, so check those before blaming the generator.

The policy version consistency test
StepInputCheckPass condition
1. Date every caseSynthetic cases and the generation recipeEach case has an as-of value for the anchor event its rules use, set before generation, not inferred afterwardsNo case lacks an anchor date for any rule it touches
2. Resolve the governing versionVersioned rule tableJoin each case to the version valid on its anchor dateExactly one governing version per rule per case, or a recorded transition overlap
3. Extract governed fieldsCase text or recordParse the fields the rule governs, such as remedy offered or deadline statedExtraction agrees with a human-labelled sample of the same cases
4. Score against the governing versionExecutable ruleCompare extracted fields with the expected valuesCase labelled consistent, inconsistent or not applicable
5. Score against neighbor versionsPrevious and next versionsRepeat step 4 with each adjacent versionInconsistent cases matching a neighbor are labelled wrong version; matching none are labelled unexplained
6. Separate indistinguishable casesResults of steps 4 and 5Flag cases where both versions give the same answerIndistinguishable cases are excluded from the version error rate
7. Compare with the real holdoutUntouched real cases from the same periodsRun steps 2 to 6 on real cases and compare rates by periodSynthetic wrong-version rate is no higher than the real rate within the agreed margin
8. Plant negative controlsCases deliberately generated under the wrong versionConfirm the test flags themEvery planted case is caught; otherwise the extractor or rule code is fixed first

Worked example (illustrative): refund conversations on both sides of a rule change

U.S. airline refund rules offer a clean, public example of a version boundary. The Department of Transportation states that key automatic refund requirements took effect on May 16, 2024, when the FAA Reauthorization Act of 2024 was signed, and that the compliance date for other refund-related requirements in its final rule was October 28, 2024. It also states that if a consumer chooses a voucher or credit instead of a refund on or after May 16, 2024, the voucher or credit must be valid for at least five years, or a cash refund is required.

Suppose a team generates customer support conversations about cancelled flights to fine-tune a service assistant, with case dates spread across 2023 to 2025. The example is illustrative; it is not guidance on what any airline owed any passenger.

The rule table holds two versions of the voucher validity rule keyed on the date the consumer chose the voucher. Step 5 surfaces the expected failure: a share of conversations dated in early 2024 have the agent promising a voucher valid for five years, which matches the later version, not the one governing those dates. The illustrative result in this run: 300 conversations dated before May 16, 2024 touch voucher validity, 41 of them state the five-year term, and step 6 removes 9 because the case's airline had already offered a term that long under its own policy, leaving 32 wrong-version cases, about 11 percent.

Step 7 gives that number its meaning. If real 2024 support transcripts from the same carrier show agents quoting the new term early in some cases, part of the synthetic rate is faithful to reality, and the right response is to label it, not remove it. If the real rate is near zero, the generator applied today's rule to last year's case, and the fix belongs in the prompt, the retrieval index or the constraint, followed by a rerun of all eight steps.

The exception in this example is worth keeping in view. A case that is consistent with both versions proves nothing about the generator, which is why step 6 exists. Counting those cases as passes would make a generator that ignores dates look accurate.

Which boundary cases does the generated set leave out?

Usually the cases that live at the boundary: those opened under one version and resolved under the next, grandfathered cases, and the period when staff were still learning the change. Real data is full of these, and generators tend to produce clean cases that sit firmly inside one version.

Measure the gap directly. Count, in the real holdout and in the synthetic set, the share of cases whose anchor dates fall within a set window around each version change, the share whose anchor events span two versions, and the share where the real record shows the old rule applied after the change. Large differences mean the synthetic mixture underrepresents transitions, and a model trained mostly on it will handle them poorly.

Also look for policy exceptions the generator never saw: manual overrides approved by a supervisor, regional variants and remedies offered outside the written policy. These appear in real records as inconsistent with every version, and a synthetic set with no unexplained cases at all is a sign that the generator is smoother than the business it imitates.

  • Transition window share: cases within an agreed number of days of a version change, real versus synthetic.
  • Straddling share: cases whose anchor events fall under two versions.
  • Late application share: real cases where the old rule was applied after the change.
  • Unexplained share: cases consistent with no version, a proxy for overrides and local practice.
  • Version coverage: every version in the rule table has at least a minimum count of synthetic cases.

What the generation record has to disclose

Enough that a reader can tell which policy versions the generator could see and how cases were dated and filtered. The Croissant responsible AI vocabulary asks datasets to document which data sources were used, versioning with timestamps and how data was processed, and lists the method used to generate synthetic data where applicable. For policy-bearing data, extend that with the specifics below.

Disclose the grounding sources with the policy version identifiers they contained, the generator and its version, the prompts or constraints that mention policy, how as-of dates were assigned, any filtering applied after generation, the mixture ratio of synthetic to real cases, and the version consistency results by rule and period. Include the negative-control catch rate, because a test that never fires on planted errors is not evidence.

Two further disclosures protect the real side. Record that the holdout used in step 7 was never shown to the generator or used to select prompts. And screen synthetic cases for near-copies of grounding records, since a generator that memorized a real case may reproduce a customer's details along with the correct policy version.

Where the test breaks down and a domain validator has to decide

When the rule cannot be executed from the record. The test assumes the policy is written, dated and mechanical enough to evaluate from the case's fields. Discretionary policies, unwritten local practice and rules that depend on facts the record does not capture cannot be scored this way, and pretending otherwise produces false confidence.

Narrow the scope first. Run the test only on rules with dated versions and a computable outcome, and report the remaining rules as unverified rather than passing. Where versions have no reliable effective date, or the anchor event is disputed, a domain validator who knows the operation should set the anchor and sign the rule table before anything is scored.

Bring in counsel or a compliance specialist when the policy is a legal requirement, as in the airline example, because the effective dates, phase-ins and enforcement positions are matters of law that can change after publication. The test checks consistency with a rule table; it does not establish what the law required in any particular case.

Dated real cases as the anchor for synthetic checks

The real holdout is the part of this test that cannot be synthesized. It needs genuine cases with trustworthy dates spanning at least one policy change, ideally with the policy documents the business used at the time.

Teams without such cases can ask SourceX, which assembles nothing in advance and instead looks for records at U.S. businesses once a request states the scope, volume, language, format, timing and licensing needs. A request for a version-anchored holdout should also name the rules, the period and the anchor events. Before any case changes hands, the business that holds it signs an agreement settling scope and terms.

Frequently asked questions

What is policy version fidelity in synthetic data?

It is the degree to which generated cases follow the policy version that governed on each case's date, rather than the current version. It is separate from statistical fidelity, which compares distributions and can pass while every case applies the wrong rule.

Can standard synthetic data quality reports detect the wrong policy version?

Usually not. Distribution, correlation and boundary metrics compare values, not the rule that produced them, so a wrong-version outcome that falls within real ranges passes. A rule-aware check keyed on the case date is needed.

How do I assign dates to synthetic cases?

Set the as-of date for each rule's anchor event as part of the generation recipe, before the case is written, and pass it to the generator with the version of the policy that applies. Inferring dates after generation lets the generator's assumptions leak into the test.

What if both policy versions produce the same answer for a case?

Label the case indistinguishable and leave it out of the error rate. Counting it as a pass makes a generator that ignores dates look accurate.

How large should the real holdout be?

Large enough to estimate the wrong-version and transition rates for each period you care about with useful precision. Plan the counts by version and period first, because a holdout with few cases near each change cannot test the boundary that matters most.

Sources

  • LLM effective knowledge cutoffs often differ drastically from reported cutoffs and differ between resources. Source
  • The SDV library lets users define business rules in the form of logical constraints for synthetic data. Source
  • SDMetrics evaluates synthetic data by comparing it with real data on quality and privacy; BoundaryAdherence checks whether synthetic values respect the real data's min and max bounds. Source
  • dbt snapshots implement type 2 slowly changing dimensions over mutable tables, adding dbt_valid_from and dbt_valid_to columns. Source
  • DOT: key automatic refund requirements took effect May 16, 2024; other refund requirements had an October 28, 2024 compliance date; vouchers chosen on or after May 16, 2024 must be valid for at least five years or a cash refund is required. Source
  • The Croissant RAI vocabulary documents data sources, versioning with timestamps, processing, and the method used to generate synthetic data. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies