Systems and records
Synthetic data rare combination coverage without impossible workflows
By SourceX Editorial · Updated
11 sections
Short answer
To generate rare combinations without impossible workflows, list a case's factors and levels, then classify every combination as observed, feasible but unobserved, or ruled out by a cited policy or system rule. Generate only into feasible cells with the rules applied during generation, check each case's step order against the process, and evaluate the mixture on real rare cases the generator never saw.
System export
Synthetic data rare
Key takeaways
- A generator cannot distinguish a rare combination from an impossible one, so feasibility has to be written down first.
- Apply rules during generation; filtering afterward silently removes valid rare combinations along with invalid ones.
- Every exclusion rule needs a source, an effective date and an owner, or it is only an assumption.
- Measure coverage over feasible cells only, and report violations, strength and policy version with it.
- Check generated cases as step sequences with conformance checking, and prove the checks work with negative controls.
- Judge the mixture per cell on untouched real rare cases, and disclose how the synthetic share was derived.
How do you generate rare combinations without creating impossible business workflows?
List the factors that define a case and their levels, then sort every combination into one of three bins before generating anything: observed in real records, feasible but unobserved, or impossible under a written rule with a cited source. Generate only into the feasible bins, check each generated workflow step by step against the process, and judge the result on real rare cases the generator never saw.
The difficulty is that a generator cannot tell a rare combination from an impossible one. Both are absent or nearly absent from the grounding data. A language model asked for an invoice that is two-way matched and blocked for a missing goods receipt will happily write one, even though in a company that runs two-way matching no receipt is ever checked. Train on enough of those and the model learns a workflow that does not exist.
The method below borrows from combinatorial testing, where engineers have long had to cover many parameter combinations while excluding the ones a system can never produce. The asset is a feasibility constrained scenario coverage matrix: a table that records, for each combination, how often it appears in real data, whether it is allowed, which rule says so, and what to generate.
Why filtering impossible rows after generation is the wrong order
Put the feasibility rules into the generation step, not into a cleanup step afterward. Microsoft's documentation for PICT, its pairwise test generation tool, makes the argument plainly: you cannot simply remove violating test cases from the result, because an offending case may also cover other valid pairs that would then go untested, so disallowed combinations should be eliminated during generation.
The same logic applies to synthetic training data. If you generate freely and then drop impossible records, the rare but valid combinations that happened to share those records disappear with them, and the coverage you thought you had bought shrinks without anyone noticing. NIST's ACTS tool, built for the same problem, generates sets that ensure t-way coverage of input values with support for constraints, which is the property you want: coverage of every allowed combination and only allowed combinations.
There is a second reason for the order. A rule applied at generation time is written down, versioned and reviewable. A rule applied by a filter script is often an undocumented regular expression, and the next team to regenerate the data will not know it existed.
Which factors and levels belong in the input dictionary?
Include the attributes that change what a correct handler does, at the levels your real systems actually record. The input dictionary below is an illustrative accounts payable example; the factor names, levels and sources are invented to show the format, not drawn from a real company.
Keep factor counts modest. Every added factor multiplies the number of combinations, and NIST's covering array work shows that most of the value comes from covering every pair or triple of factor values rather than every full combination. Choose the strength, two-way or three-way, before you build the matrix and record it.
| Factor | Levels | Where the levels come from | Why it changes the workflow |
|---|---|---|---|
| Invoice channel | EDI, emailed PDF, supplier portal | Distinct values in the AP system's source field | Channel decides which fields arrive structured and which need extraction |
| Match type | two-way, three-way | Purchasing policy and the PO's match setting | Three-way matching adds a goods receipt check that two-way never runs |
| Exception | none, price variance, quantity variance, missing receipt, duplicate suspect | Hold reason codes in the AP system | Each hold has its own resolution path and owner |
| Vendor status | active, on hold, newly onboarded | Vendor master status field | Held and new vendors route to extra review |
| Approval route | auto-approve, manager, controller | Approval workflow configuration | Determines who acts and how long the case waits |
| Policy version | Policy effective dates in force during the record period | Dated policy documents | Rules change, so a combination valid last year may be invalid now |
Writing feasibility rules a reviewer can trace to a source
Every rule that removes or restricts a combination needs a source, an effective date and an owner. A rule nobody can trace is an assumption, and the matrix should show it as one.
At a glance
Writing feasibility rules a reviewer can trace to a source
Write rules in a form a tool can enforce. PICT, for example, accepts conditional constraints such as IF [Match type] = "two-way" THEN [Exception] <> "missing receipt". The SDV synthetic data library similarly lets users express business rules as logical constraints. Whatever the tool, keep the human-readable rule, its citation and the machine form side by side in one register.
- Hard rule: the system or policy makes the combination impossible. Cite the configuration screen, policy clause or system behavior. Example, illustrative: two-way matched invoices never carry a missing receipt hold.
- Conditional rule: possible only when another factor takes a specific value. Example, illustrative: controller approval applies only above a policy threshold, so auto-approve plus duplicate suspect is excluded.
- Ordering rule: the steps must happen in sequence. Example, illustrative: a vendor cannot be paid before onboarding is complete.
- Effective-dated rule: valid only inside a policy period. Record both dates and generate per period rather than mixing versions.
- Soft rule: allowed but rare, documented by a few real cases. Do not exclude it; mark it so generation can target it deliberately.
- Unknown: nobody can say whether the combination is possible. Do not generate it; send it to a domain validator.
The feasibility constrained scenario coverage matrix
The matrix is one row per factor combination at the chosen strength, with five decisions recorded for each. The rows below are an illustrative two-way slice of the accounts payable example; real counts would come from the grounding records, and these are placeholders labeled as such.
Status values are fixed: observed, feasible-unobserved, soft-rare, infeasible and unknown. Generation decisions follow from status, not from taste. Infeasible and unknown cells are never generated, and soft-rare cells are generated only with a recorded ceiling relative to the real count.
| Combination | Real count, illustrative | Status and rule | Generation decision |
|---|---|---|---|
| Three-way match with missing receipt | Observed in 140 grounding records | Observed; no rule needed | Generate variants for diversity; keep real cases in the holdout |
| Two-way match with missing receipt | 0 | Infeasible; hard rule from the purchasing policy match setting | Never generate; use as a negative control |
| Supplier portal with duplicate suspect | 3 | Soft-rare; duplicate check runs on every channel | Generate up to a recorded multiple of real cases and label as synthetic |
| Newly onboarded vendor with auto-approve | 0 | Infeasible; ordering rule from vendor onboarding policy | Never generate |
| EDI with quantity variance | 0 | Feasible-unobserved; no rule excludes it and an AP lead confirms it can occur | Generate a small set and flag for domain review |
| On-hold vendor with controller approval | 0 | Unknown; policy silent | Do not generate; escalate to domain validator |
| Emailed PDF with price variance | Observed in 610 grounding records | Observed; no rule needed | No generation needed at two-way strength |
How should coverage be measured before and after generation?
Measure the share of feasible combinations that the data covers, and never count infeasible cells in the denominator. Counting them makes a dataset that respects the rules look incomplete and rewards a generator for producing nonsense.
At a glance
How should coverage be measured before and after generation
NIST's combinatorial coverage measurement work, described in NISTIR 7878, defines ways to judge how much t-way coverage any existing set has, whether or not it was designed for coverage, so the gaps can be found and filled. A NIST conference paper by Lanus, Freeman, Kuhn and Kacker extends that idea to machine learning, using combinatorial coverage metrics to compare training and test datasets. The steps below apply both ideas to a synthetic mixture.
- Compute the feasible cell set from the matrix: all combinations at the chosen strength minus infeasible and unknown cells.
- Real coverage: share of feasible cells with at least one real grounding record.
- Mixture coverage: share of feasible cells with at least one real or synthetic record. The gap between the two is what synthesis added.
- Violation count: number of generated records that fall into infeasible cells. The target is zero; any non-zero value means a rule failed to reach the generator.
- Holdout difference: share of combinations in the real holdout that never appear in the training mixture. This is the cell list the mixture still fails to represent.
- Report each figure with the strength used and the policy version, since both change the result.
Checking each generated case as a sequence of steps
Attribute combinations can all be valid while the generated workflow is still impossible, because the steps happen in the wrong order or a required step is skipped. Check sequences as well as attributes.
Process mining tools already do this for real event logs. pm4py's conformance checking compares a process model with an event log using token-based replay, alignments or footprints, and reports fitness and precision. Run the same check on generated cases converted to event logs: a synthetic invoice that is paid, then approved, then matched should fail replay against the AP process model, and the failure should block it from the training set.
Add negative controls so you know the checks work. Deliberately generate a small batch of cases from infeasible cells and with steps out of order, and confirm the validator rejects every one. PICT's guidance on negative testing offers a useful discipline here: include only one invalid value per negative case, so one error cannot mask another and you can see which rule caught it.
Judging the mixture on real rare cases the generator never saw
Set aside real rare cases before any generation starts and never show them to the generator, the prompt writer or the person tuning rules. scikit-learn's guidance on leakage states the principle in general form: test data should never be used to make choices about the model. Here the model includes the generator and its rules.
Compare per cell, not only overall. SDMetrics, for instance, evaluates synthetic data by comparing it with the real data it mimics, including whether synthetic values respect the real minimum and maximum. Apply such checks within each matrix cell, then train on the mixture and test on the untouched real holdout, reporting results per cell alongside the real count that supports each one.
Read the failures as data. A cell where the model trained on the mixture does worse than the model trained on real data alone usually means the generator captured the attribute combination but not the handling that goes with it, such as which approver signs a quantity variance from a held vendor.
Illustrative scenario: filling rare exception paths in an AP dataset
This scenario is illustrative; every count and percentage is invented to show the arithmetic. A team holds a real accounts payable sample covering five of the six factors above. With three channels, two match types, five exceptions, three vendor statuses and three approval routes, two-way strength gives 100 value pairs; rules mark 19 infeasible and 4 unknown, leaving 77 feasible cells. Real records cover 50 of them, so real coverage is 65%.
The team generates only into the 27 uncovered feasible cells and the soft-rare cells. Mixture coverage rises to 77 of 77, or 100% of feasible cells, and the violation count is zero. Conformance checking rejects 7% of generated cases for step order, almost all of them paying before approval, which leads to a fix in the generation prompt rather than a filter.
The holdout of 120 real rare cases tells a less tidy story. The model trained on the mixture improves on the cells real data already covered and on most new cells, but does worse on EDI with quantity variance, a feasible-unobserved cell. The domain validator reviews it and finds that EDI quantity variances in this illustrative company are resolved by the receiving team, not AP, a handling rule no factor captured. The team adds a resolving team factor and regenerates.
The exception matters as much as the gain. Synthetic coverage of a cell is a claim that the attributes appear together; it is not evidence that the handling is right, and only real cases can supply that.
Which assumption moves the result, and what to disclose
Three assumptions change the decision most: the strength you chose, the policy version the rules encode, and how strictly you classed borderline cells. Show uncertainty by re-running the coverage figures under each alternative and reporting the range, and by publishing the real count beside every generated cell so readers can see which results rest on a handful of real cases.
NIST's covering array guide notes that efficient generation does not solve the oracle problem, meaning you still need a way to know the right outcome for each case. That is why unknown cells go to a domain validator, and why a cell whose real count is zero should never be the only evidence for a capability claim.
Disclose the derivation with the dataset: the grounding records and their period, the generator and its version, the rule register with sources and dates, the share of synthetic records per cell, the selection or filtering steps, and the known mismatch between the mixture and the real distribution. Stop and bring in specialist review when rules depend on law or regulated policy, when unknown cells outnumber feasible-unobserved ones, or when the holdout shows the mixture underperforming real data on a cell you plan to rely on.
Sourcing the real rare cases that anchor the matrix
The matrix is only as good as the real records behind its observed and soft-rare cells and its holdout. When those are thin, the fix is more real exceptions, not more generation. Teams short of real exceptions can ask SourceX to search U.S. businesses for them, since it holds no datasets in inventory, and the request can list the factors, levels and rare cells to fill alongside volume, format, timing and licensing terms. Any business that supplies those exceptions agrees the scope and terms in writing before handing anything over.
Frequently asked questions
What is a rare combination in synthetic data?
It is a set of attribute values that can occur together in the real process but appears in few or no grounding records. Rare combinations are where synthetic data can add coverage, and also where generators most often produce cases the process could never create.
How do I know whether a missing combination is impossible or just rare?
Look for a rule that excludes it in policy, system configuration or process documentation, and cite it. If there is no rule and a domain expert confirms it can happen, treat it as feasible but unobserved. If nobody can say, mark it unknown and do not generate it.
Should synthetic rare cases be used in the evaluation set?
No. Keep evaluation on real cases the generator never saw. Synthetic cases can be useful negative controls for validators, but scoring a model on generated cases mostly measures how well it learned the generator.
What should a feasibility constrained scenario coverage matrix include?
One row per factor combination at a chosen strength, the real count, a status from a fixed list, the rule and its source that set the status, and the generation decision. Keep the input dictionary and the rule register with it.
Is pairwise coverage enough for business workflows?
It is a sensible starting point because it keeps the matrix small. Move to three-way coverage for factors known to interact, such as match type, exception and approval route, and record the strength used with every coverage figure.
Sources
- Microsoft's PICT documentation says violating combinations should be eliminated during generation with constraints, because deleting offending rows afterward can lose valid pairs; it also describes seeding and one-invalid-value negative testing. Source
- pm4py's conformance checking compares a process model with an event log, using token-based replay, alignments or footprints, and reports fitness and precision. Source
- The SDV library lets users define business rules as logical constraints for synthetic data and generate a quality report. Source
- SDMetrics evaluates synthetic data by comparing it with the real data, including a check of whether synthetic values respect the real data's min and max bounds. Source
- scikit-learn's guidance on data leakage says test data should never be used to make choices about the model and that data should be split before preprocessing. Source
- NIST SP 800-142 describes a covering array as a test suite that covers all t-way combinations of parameter values, and notes that the oracle problem remains. Source
- NISTIR 7878 defines measures of the t-way combinatorial coverage of any existing test set, so gaps can be found and filled. Source
- A NIST conference paper by Lanus, Freeman, Kuhn and Kacker introduces combinatorial coverage metrics for comparing machine learning training and test datasets. Source
- NIST's ACTS tool generates test sets that ensure t-way coverage of input parameter values, with support for constraints and variable-strength tests. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.