AI uses for records
Synthetic data business rule consistency: an invariant test suite
By SourceX Editorial · Updated
10 sections
Short answer
To validate business-rule consistency across a synthetic dataset, write the simulated company's rules as executable invariants and count violations at four levels: inside one record, across linked records, along each document's lifecycle and in ledger or inventory totals. Test rules the generator was never told about, seed known violations to prove each test fires, and run the same suite on untouched real records to learn what violation rate to expect.
Checklist
Synthetic data business rule
Key takeaways
- Column-level validity and fidelity scores check ranges, categories and keys; they do not check arithmetic or ordering across fields and tables, which is where synthetic business records usually break.
- Every invariant needs a written source and an effective date, because business rules change and a generator grounded in old policy will produce records that were valid last year.
- A rule the generator was constrained to obey proves nothing when it passes; the informative tests are the rules it was never given.
- Seed known violations into a copy of the data before trusting any test, so a zero count means clean data rather than a broken query.
- Real company data breaks some rules legitimately; run the suite on an untouched real holdout and treat a synthetic set that is cleaner than reality as a warning, not a win.
How do you validate business-rule consistency across a synthetic dataset?
Turn each business rule the simulated company must follow into an invariant, a statement that should be true of every record or group of records, and write it as a test that returns the records breaking it. Count violations at four levels: within a single record, across linked records, along a document's lifecycle over time, and in totals such as a ledger or inventory balance. Then run the identical suite on an untouched set of real records, so each synthetic violation rate can be read against what the real business actually produces.
The goal is narrower than overall synthetic data quality. A dataset can match real distributions closely and still contain invoices whose lines do not add up to the total, shipments dated before their orders and payments against invoices that were canceled. A model trained on those records learns a company that cannot exist, and an agent evaluated on them is graded against impossible states.
Why column-level quality scores pass rule-breaking records
Most automated synthetic data reports test columns and keys, not rules. In SDMetrics, the open-source library from the SDV project, the single-table data validity property applies boundary adherence to numeric and datetime columns, category adherence to categorical columns and key uniqueness to primary keys, and the multi-table relationship validity property adds referential integrity and cardinality boundary checks. Those are useful, and they will catch a shipment that points to an order that does not exist. They will not catch a shipment of 12 units against an order for 10, because each value on its own sits inside the real range.
NIST's SDNist report tool, used in the NIST Collaborative Research Cycle on deidentified data, makes the same point for census-style data. Its report describes deterministic dependencies between features, gives the example of a generator producing toddlers with PhDs, calls such records inconsistencies, and says plainly that its own check is not exhaustive in the way a production-grade system should be. Business data has far more of these dependencies than demographic tables do, and no generic list covers a particular company's rules. The suite has to be written for the company being simulated.
Writing a simulated company's rules down, with their source and date
Each invariant should carry four things: the rule in plain words, where it comes from, the date range it applies to and whether it is hard or soft. A hard rule can never be broken in a correct record, such as a journal entry whose debits equal its credits. A soft rule usually holds but has legitimate, recorded exceptions, such as a payment arriving before an invoice because the customer prepaid.
At a glance
Writing a simulated company's rules down, with their source and date
- Accounting identities: balanced entries, invoice totals equal to lines plus tax, credit memos tied to an original invoice.
- System validation rules: what the ERP or order system refuses to save, taken from its configuration, not from memory.
- Policies and contracts: credit limits, approval thresholds, return windows and price agreements, each with the date it took effect.
- Process order: the sequence a document moves through, from order to pick, ship, invoice and payment, and which steps can be skipped.
- Physical limits: stock cannot be shipped that was never received, unless the system explicitly allows negative inventory.
- Known exceptions: overrides, manual journals and backdated entries the real business permits, with the evidence each leaves behind.
The invariant validation suite for a simulated distributor
The table below is a starting suite for a simulated wholesale distributor with customers, orders, shipments, invoices, payments, credit memos and a general ledger. Adapt the rules to the company being simulated; the structure carries over to service, manufacturing and subscription businesses with different documents.
Scope each invariant as tightly as the business does. I1 needs the rounding rule the company actually uses, at line or at header. I5 needs the overship tolerance from the customer agreement, which may differ by customer. I10 needs the credit limit that applied on each date, not the current one. An invariant written more loosely than the real rule will pass records the business would have rejected, and one written more strictly will flag legitimate records as generator errors.
| Invariant | Level | Hard or soft | Failing-row test |
|---|---|---|---|
| I1 Line amount equals quantity times unit price less discount | Single record | Hard | Order lines where the computed amount differs beyond rounding |
| I2 Invoice total equals its lines plus tax | Linked records | Hard | Invoices whose header total differs from the sum of their lines |
| I3 Every journal entry balances | Linked records | Hard | Entries where total debits differ from total credits |
| I4 Shipments, invoices and payments reference existing parents | Linked records | Hard | Child rows with a non-null parent key that has no match |
| I5 Shipped quantity never exceeds ordered quantity | Linked records | Soft | Order lines shipped above quantity with no logged tolerance |
| I6 Ship date on or after order date | Lifecycle | Hard | Shipments dated before their order |
| I7 Payment on or after invoice date | Lifecycle | Soft | Payments dated before their invoice with no prepayment flag |
| I8 No shipment or invoice on a canceled order | Lifecycle | Hard | Documents created after the order's cancellation time |
| I9 Credit memo does not exceed its original invoice | Linked records | Hard | Memos larger than, or missing, the invoice they reverse |
| I10 Open receivables stay within the credit limit in force | Totals | Soft | Customers over limit on a date with no approved override |
Running each invariant as a query that returns failing rows
Write every invariant as a query that selects the records breaking it, so a passing test returns zero rows and a failing test hands the reviewer the exact records. This is how dbt, a widely used data transformation tool, defines data tests: each is a select statement that looks for records disproving an assertion, and the test passes when it returns no rows. Its singular tests are one-off SQL files for exactly this kind of company-specific rule.
Two details from the same documentation matter for synthetic data. dbt's relationships test, its referential integrity check, skips null values the way database foreign keys do, so a synthetic shipment with no order key at all passes it; pair it with a not-null test wherever the parent is required. And a generic test applied to many columns is better than ten nearly identical singular ones, because the suite will be rerun on every new generator version.
Report each invariant as a count, a rate per thousand relevant records and a small sample of failing rows. Rates make generator versions comparable; samples let a domain validator see whether the failures are one bug or many.
Testing rules the generator was never told about
Some generators accept rules as constraints. The SDV library, for example, has an Inequality constraint that keeps one column greater than another and a FixedCombinations constraint that limits columns such as country and city to combinations seen in the real data. Constraints are a good way to stop obvious errors, but they change what a passing test means. A rule the generator was forced to satisfy will pass whether or not the generator learned anything about the business.
Split the suite in two and report the halves separately. Enforced invariants are a regression check: they should pass at exactly zero, and any failure is a pipeline bug. Held-out invariants, rules deliberately not given to the generator, are the real test of whether the synthetic company behaves like a company. Choose held-out rules that depend on several fields or tables, such as I7 or I10, because those are the dependencies a generator is least likely to pick up by accident.
Then add negative controls. Copy a sample of the synthetic data, inject a known number of violations for each invariant, and confirm each test finds exactly those. A test that reports zero violations on data you deliberately broke is a broken test, and without this step a clean report cannot be told apart from a query that never matches anything.
Worked example: order-to-cash records from a simulated distributor (illustrative)
This example is illustrative; the counts and percentages are invented to show the method. A team generates one simulated year of order-to-cash records for a distributor and holds out three months of real records from the business whose data grounded the generator, never shown to it.
Enforced invariants I1, I4 and I6 pass at zero, as they should. A negative control finds that the first version of I3 compared debit and credit totals per day rather than per entry and missed 40 of 40 injected unbalanced entries; after the fix it finds all 40.
Held-out invariants tell the real story. I2 fails on 6 percent of synthetic invoices, because the generator produced invoice headers and lines from separate learned distributions. I8 fails on 2 percent: shipments created after their order was canceled. Neither appears in the real holdout at all. Both are generator defects, and the fix is structural: generate lines first and derive the header, and generate status changes as a sequence rather than as independent fields.
The exception is I7. The synthetic set shows payments before invoice on 0 percent of records; the real holdout shows them on about 3 percent, every one with a prepayment flag on the customer. The synthetic data is cleaner than reality here, and that is the failure. A model trained on it would treat every early payment as an error. The team adds prepaid customers to the grounding set and checks that the rate moves toward the real one.
Which legitimate rule breaks has the simulated company smoothed away?
Rules hide the exceptions that real businesses live with, so a synthetic set that obeys every rule perfectly is usually missing something. Compare each soft invariant's synthetic violation rate with the real holdout's, and look hardest where the real rate is above zero and the synthetic rate is zero.
At a glance
Which legitimate rule breaks has the simulated company smoothed away
Put the comparison in the disclosure that ships with the dataset: the suite version, which invariants were enforced and which were held out, the synthetic and real violation rate for each, how large the real holdout was and the date range of the policies encoded. A buyer reading that table can see at once which exceptions the synthetic company reproduces and which it has smoothed away, and can decide whether to mix in real records for the classes it lacks.
- Approved overrides: credit holds released by a manager, price exceptions, shipments above order quantity within a tolerance.
- Corrections: reversals, credit and rebill, manual journal entries after period close and the audit trail each leaves.
- Policy changes: a credit limit or approval threshold that changed mid-year, so records on either side follow different rules.
- Rare combinations: a return on a partially shipped, partially paid order, which few generators produce unless asked.
- Cross-entity effects: one customer's default holding up shipments to a related account.
Limits of a clean invariant report, and who signs off
A clean invariant report shows that the synthetic records are internally coherent, not that they are useful or safe. Usefulness is a separate test: the train-on-synthetic, test-on-real protocol proposed by Esteban, Hyland and Rätsch trains a model on generated data and scores it on held-out real examples. Privacy is separate again, because a generator can produce perfectly consistent records by copying real ones. The invariant suite sits beside those checks; it does not replace them.
Stop and bring in a domain validator, usually the controller or operations lead of the business being simulated, when the rules cannot be traced to a system configuration or policy, when held-out invariants fail above a level the team agreed in advance, or when no untouched real holdout exists to set expected rates. Rerun the suite whenever the generator, its grounding records, the synthetic-to-real mixture or any encoded policy changes, and publish the suite version and date with the dataset so a buyer can see exactly what was checked.
Getting real records to set the expected violation rates
The expected rate for every soft invariant comes from real records, so the quality of a synthetic company depends on having a real company to compare it with. Those records need the fields that make the rules testable: document links, status histories with timestamps, override and approval flags, and the dated policies in force.
Real order-to-cash records for that comparison can be requested from SourceX, which sources them from U.S. businesses to each buyer's stated scope, volume, format, timing and licensing needs rather than from stock. A useful holdout typically covers two years or more and tens of thousands of documents in one system, and each request is assessed on its own.
Frequently asked questions
What is a business rule invariant in synthetic data?
It is a condition every correct record, or group of linked records, must satisfy, such as an invoice total equal to its lines plus tax or a shipment dated after its order. Writing rules as invariants turns them into tests that count violations instead of opinions about realism.
Do synthetic data quality reports check business rules?
Generally only partly. Standard reports check column ranges, categories, key uniqueness, referential integrity and distribution similarity. Rules that combine several fields or tables, such as balanced journal entries or credit limits, have to be written as custom tests for the business being simulated.
Should synthetic data have zero business rule violations?
Only for hard rules that no correct record can break. Soft rules have legitimate exceptions in real data, such as prepayments or approved overrides, and the synthetic violation rate should look like the real one. Zero where the real rate is above zero means the generator erased an exception.
Can generator constraints replace an invariant test suite?
No. Constraints stop known errors during generation, but a rule the generator was forced to obey will always pass. Keep enforced rules as a regression check and test the business against rules the generator never saw.
How many real records are needed to set expected violation rates?
Enough to observe the rarer soft-rule exceptions more than a handful of times, which depends on how rare they are. If a real exception appears only a few times in the holdout, report the comparison with its uncertainty rather than as a pass or fail.
Sources
- SDMetrics' single-table data validity property applies boundary adherence to numeric and datetime columns, category adherence to categorical columns and key uniqueness to keys. Source
- SDMetrics' multi-table relationship validity property uses referential integrity and cardinality boundary adherence metrics. Source
- NIST's SDNist report describes deterministic dependencies between features, calls failures such as toddlers with PhDs inconsistencies and says its check is not exhaustive. Source
- dbt data tests are select statements that return failing records and pass when no rows are returned; singular tests are one-off SQL assertions. Source
- dbt's relationships test checks referential integrity and excludes null values, so a separate not_null test is needed when nulls should fail. Source
- SDV's Inequality constraint ensures one column is greater than another in generated data. Source
- SDV's FixedCombinations constraint keeps combinations across columns, such as country and city, to valid ones. Source
- Esteban, Hyland and Rätsch proposed train on synthetic, test on real: train a model on generated data and test it on held-out real examples. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.