Skip to content

AI uses for records

Post-training data generalization: did the model learn the workflow or one company's rules?

By SourceX Editorial · Updated

10 sections

Short answer

To test whether a model learned a workflow or memorized one company's rules, hold out whole companies rather than random records. Train on some companies, evaluate on companies the model never saw, and tag each test item as a shared workflow step or a local rule. A model that learned the workflow improves on shared steps at unseen companies; one that memorized carries a training company's thresholds into cases where they are wrong.

Evaluation split

Post training data

This page covers post-training data generalization: did the model learn the workflow or one company's rules.Illustrative

Key takeaways

  • Random record splits reward a model for recognizing a company, so post-training data generalization must be tested on companies held out entirely.
  • Tagging each evaluation item as a shared step, a local rule or a rule conflict turns one accuracy number into an answer about what was learned.
  • Policy bleed, a training company's rule applied where it does not hold, is the clearest sign of memorized local rules.
  • Choose checkpoints on a validation company that is neither trained on nor evaluated, and keep budgets identical across arms.
  • A planted canary rule measures how readily the model absorbs rule text seen only a few times.

How do you tell a learned workflow from one company's memorized rules?

Hold out whole companies, not random records. Train on records from some companies, evaluate on companies the model never saw, and split every test item into steps the workflow shares across companies and steps a single company's policy decides. A model that learned the workflow improves on the shared steps at the unseen companies. A model that memorized one company's rules carries that company's thresholds, approvers and wording into places where they are wrong.

The distinction matters because post-training data is usually bought or licensed to fix a specific failure, such as a support agent that escalates too late or an accounts payable assistant that routes exceptions to the wrong queue. Business records encode two things at once: the general procedure (match the documents, check the tolerance, route the exception, record the decision) and the local policy (this company's tolerance, this company's approval chain, this company's vocabulary). Both show up as correct labels in training, and an ordinary test set drawn from the same companies rewards both equally.

The test below separates them. It needs three or more source companies, a tag on each evaluation item that says whether the correct answer depends on a local rule, and a decision about what counts as success written down before anyone trains.

Why a random record split hides company-specific memorization

A random split puts records from the same company on both sides of the line, so a model can score well by recognizing the company rather than doing the task. The scikit-learn documentation describes the same trap for grouped data: when samples come in dependent groups, a model flexible enough to learn group-specific features can fail on new groups, and the fix is to make sure no group in the validation fold appears in the paired training fold. Benchmarks built for distribution shift make the group explicit: WILDS attaches domain metadata to each example, such as which hospital a patient's data came from, so evaluation can be grouped by that domain.

Business records are strongly grouped. One company's tickets share templates, macros, signature blocks, product names and policy phrases. A held-out ticket from that company is often a near copy of a training ticket. Lee and colleagues found that train-test overlap affected over 4% of the validation examples in standard language datasets, and that such overlap leads to overestimated accuracy. Inside a single company's records, templated text makes overlap more likely, not less.

Exact-match filtering is not enough on its own. The EleutherAI evaluation harness documentation notes that evaluations test generalization and are less useful when test data also exists in the training set, and it describes flagging contaminated test documents by n-gram overlap so a clean score can be reported. Run that kind of overlap check across companies too, because franchise systems, shared software vendors and common form letters create cross-company duplicates.

The cross-company generalization test, arm by arm

The asset is a set of training and evaluation arms that each isolate one explanation for a score change. Run them with the same base model, the same training budget and the same frozen evaluation items. The leave-one-company-out arm is the core; the others tell you why it moved.

Rotate the held-out company so each one takes a turn as the evaluation company, and report the spread across rotations as well as the mean. One unusual company can dominate a single split: a firm with far stricter policies, older software or a different customer mix will look like a generalization failure even when the workflow transferred to every other company. If one rotation is an outlier, read its items before averaging it away, because that company may be closer to the deployment you care about than the rest of the pool.

Keep the evaluation items frozen from the first run. Adding items after seeing results, or relabeling items the model got wrong, quietly turns the test set into a second training signal.

The cross-company generalization test, arm by arm
ArmTraining dataEvaluated onWhat it isolates
A. BaselineNo new examples; current modelAll held-out items at all companiesWhere the failure stands before any data is added
B. In-companyCandidate examples from companies 1 to kHeld-out records from the same companiesThe optimistic score a random split would report
C. Leave one company outCandidate examples from all companies except one, rotatedThe excluded company's itemsTransfer of the workflow to a company the model never saw
D. Shared-step sliceSame as COnly items tagged shared stepWhether the general procedure improved
E. Rule-conflict probesSame as CItems where a training company's rule gives the wrong answer for the targetPolicy bleed: training rules applied where they do not hold
F. Canary ruleC plus one invented rule inserted at a set frequencyPrompts that would trigger the invented ruleDirect memorization of rule text, separate from task skill
G. Regression suiteSame as CExisting capability tests unrelated to the failureDamage the new data did to what already worked

Tagging every test item as a shared step, a local rule or a conflict

Every evaluation item needs a label saying whether its correct answer is set by the shared workflow or by the company's own policy. Without that tag, a single accuracy number mixes the two and the test cannot answer its question. Tag items before training, from the source system's own policy documents, and have someone who did not build the training set check a sample of the tags.

Steps

Tagging every test item as a shared step, a local rule or a conflict

Record these fields for each item so the result can be audited and rerun:

Checklist · 7 items

  1. item_id and company_id, with company_id never shared between a training fold and its evaluation fold
  2. step_type: shared step, local rule or rule conflict
  3. rule_reference: the policy, threshold or approval chain that decides the answer, quoted from the company's own documentation or configuration
  4. expected_answer under the target company's rules
  5. bleed_answer: what a model applying a training company's rule would output instead, filled in for rule-conflict items
  6. failure_cohort: the named failure the purchase is meant to fix, so improvement can be read per cohort
  7. source lineage: system, export date and record identifiers, so any item can be traced back and removed

0 of 7 ticked. Tick each item as you go.

Fixing the selection rule, budget and model choice before training

Write the selection rule, the training budget and the model-selection method down before training, and keep them identical across arms. If arm C gets more steps, more tokens or a different learning rate than arm B, the comparison measures compute rather than data.

Model selection deserves its own decision. The DomainBed project lists three selection criteria: a random subset of the training domains, a held-out domain that is used for neither training nor testing, and an oracle subset of the test domain. Gulrajani and Lopez-Paz argue that a domain generalization method without a model selection strategy is incomplete, because a validation set drawn from the test distribution is not normally available. For a cross-company test, that means choosing checkpoints and hyperparameters on a validation company that is neither in training nor in the final evaluation. Choosing them on the evaluation company is the oracle setting and inflates the result.

The same paper found that plain empirical risk minimization, carefully tuned, performed about as well as specialized domain generalization algorithms across its benchmarks. In practice, that points the effort at the data and the split rather than at clever training tricks: which companies are in the pool, how varied their policies are and whether the evaluation companies are truly unseen.

Reading the result: fixed cohorts, policy bleed and regressions

Read the result as three numbers, not one: improvement on shared steps at unseen companies, the rate of policy bleed on rule-conflict items, and the change on the regression suite. Report each per failure cohort with a confidence interval across company rotations, since the number of companies, not the number of records, usually limits precision.

Typical patterns and what they mean:

  • Large gain in arm B, small gain in arm C: the data mostly taught company recognition. Look for templated text and near duplicates before buying more of the same.
  • Gain in arm C on shared steps, low bleed on conflict items: the workflow transferred. This is the result that justifies scaling the source.
  • Gain in arm C on shared steps, high bleed: the model learned the procedure and also the training companies' thresholds. Add rule context to the prompt or input, or add more companies with differing rules, before scaling.
  • No gain in arm C on any slice: the examples do not fix the named failure outside their own company. Narrow the scope or stop.
  • Any arm with a regression beyond the noise band measured on repeated baseline runs: treat as a blocker regardless of the gain.

Illustrative case: invoice exception routing at five distributors

The following case is illustrative; the companies, counts and percentages are invented to show how the arms read, not results from any study. A model developer wants an accounts payable assistant to route invoice exceptions correctly. The failure cohort is invoices whose price differs from the purchase order. It licenses exception histories from five distributors. All five follow the same three-way match workflow, comparing purchase order, receiving record and invoice, but each sets its own price tolerance and its own approver for overruns.

Arm B reports a large gain in correct routing on held-out invoices from the same distributors. Arm C, rotating each distributor out in turn, shows a smaller but real gain on shared-step items such as identifying which document disagrees. On rule-conflict items, roughly a third of the model's answers apply a training distributor's tolerance to the held-out distributor. The regression suite on unrelated finance questions is flat within the baseline noise band.

Reading: the workflow transferred, and the thresholds leaked. The team adds the target company's tolerance as an input field rather than leaving it to be learned, reruns arms C and E, and the bleed rate falls on the rerun while the shared-step gain holds. The exception history became useful once the local rule was supplied as context instead of being taught as a fact about the world.

When learning one company's rules is the goal

Sometimes memorizing one company's rules is the purpose. A model fine-tuned for a single customer, deployed only inside that customer's systems, should apply that customer's tolerances and approval chain exactly. For that deployment, swap the question: score rule fidelity on the customer's own held-out records, split by time so later policy changes are tested, and check that the customer's rules do not appear when the same base model serves anyone else.

Two other cases need a narrower test. With records from only one or two companies, a leave-one-company-out arm is impossible or meaningless; use a time split plus rule-perturbation items, where a policy value is changed in the input and the expected answer changes with it. And when local rules are legally required, as in regulated approvals, treat any bleed as a defect to fix in the input design rather than a tolerable error rate. Bring in the evaluation lead and, where rules have legal force, compliance owners before reading the result as a go.

Measuring memorization directly with a planted canary rule

A canary rule measures memorization of rule text directly, separate from task skill. Invent a rule that exists at no real company, such as a fictitious approval code, insert it into a small number of training examples at a known frequency, and then prompt the trained model with situations where only a model that memorized the rule would produce it.

This borrows the method of The Secret Sharer, in which Carlini and colleagues insert randomly chosen canary sequences into training data a varying number of times and measure an exposure score from the model's perplexity. They report that unintended memorization appears early in training and is not simply a symptom of overfitting. For business rules, the practical reading is that a held-out score near the training score does not prove the model avoided memorizing; the canary tells you how readily it absorbs a rule seen a handful of times.

Keep canaries out of production checkpoints, log their insertion frequency in the training manifest, and run the same probes on the baseline model so false positives from ordinary language patterns are visible.

Sourcing records from several companies for the held-out test

The test only works with records from several companies whose policies genuinely differ, and that is often the hardest input to find. Ask suppliers for the policy documents behind the records as well as the records themselves, because tagging rule-conflict items depends on knowing each company's thresholds and approval chains.

A buyer can use SourceX's request-data form to describe the workflow, the number of companies needed and the policy documentation required, and SourceX sources the records from U.S. operating businesses on request rather than holding stock. Every contributing company approves scope and terms and signs before its records reach SourceX, which then runs rights review, preparation and delivery. That makes it possible to plan a held-out company before the first delivery rather than discovering afterward that every record came from one firm.

Frequently asked questions

How many companies do you need for a cross-company generalization test?

Three is the practical minimum: at least two for training and one held out for evaluation, with a separate validation company if you also select checkpoints. More companies narrow the confidence interval more than more records per company do, because the company is the unit being generalized to. If policies are very similar across the pool, add companies with deliberately different rules.

Is leave-one-company-out the same as GroupKFold?

It is the same idea applied with company as the group. Scikit-learn's LeaveOneGroupOut holds out one group per split, and GroupKFold with as many splits as groups gives the same folds. The important part is that no company appears on both sides of any split.

What is policy bleed in a fine-tuned model?

Policy bleed is when a model applies a rule learned from one organization's records to a case governed by a different organization's rule, for example using one company's price tolerance on another's invoice. It is measured on rule-conflict items, where the training rule and the target rule give different answers.

Can deduplication alone show the model generalized?

No. Deduplication removes near copies so the score is not inflated by overlap, but a model can still learn company-specific shortcuts from distinct records. You still need held-out companies and items tagged by step type.

Should company names be removed from post-training data?

Removing names and identifiers helps privacy and reduces one shortcut, but it does not stop a model from learning a company's thresholds or phrasing. Test transfer with the held-out-company arms whether or not names are present.

Sources

  • Scikit-learn's cross-validation guide says grouped samples break the independence assumption and that validation folds should contain only groups absent from the paired training fold; LeaveOneGroupOut holds out one group per split. Source
  • DomainBed lists three model selection criteria: a subset of the training domains, a held-out domain used for neither training nor testing, and an oracle subset of the test domain. Source
  • Gulrajani and Lopez-Paz report that carefully implemented ERM matches domain generalization algorithms on their benchmarks and argue that a method without a model selection strategy is incomplete. Source
  • The WILDS benchmark of in-the-wild distribution shifts records domain metadata such as which hospital a patient's data came from and provides grouping tools for domain-aware evaluation. Source
  • The EleutherAI evaluation harness documentation says evaluations are less useful when test data also exists in training and describes n-gram overlap flagging to report a clean score. Source
  • Lee and colleagues report that train-test overlap affects over 4% of the validation sets of standard language datasets and leads to overestimated accuracy. Source
  • The Secret Sharer inserts canary sequences into training data a varying number of times, measures exposure from perplexity, and reports that unintended memorization occurs early and is not due to overfitting. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies