Skip to content

AI uses for records

Cross-company dataset generalization: test held-out suppliers before scaling

By SourceX Editorial · Updated

9 sections

Short answer

Before scaling a data purchase, test cross-company dataset generalization by holding out whole supplier organizations: train on all but one, score on the one left out, and repeat for each. Compare that held-out gain with the gain on familiar suppliers and on your own records, with intervals that count organizations, not records. Scale only if the gain survives unseen companies; otherwise buy breadth, not volume.

Evaluation split

Cross company dataset

This page covers cross-company dataset generalization: test held-out suppliers before scaling.Illustrative

Key takeaways

  • A random record split rewards a model for memorizing the supplying companies; hold out whole organizations instead.
  • The transfer ratio, held-out gain divided by in-supplier gain, shows how much of a pilot's headline travels to unseen companies.
  • Organizations are the unit of uncertainty: six suppliers give six numbers, and the interval should look that wide.
  • When the interval straddles the threshold, the next order is more companies in the pilot, not more records from the same ones.
  • Keep the buyer's own records sealed until the supplier folds pass; they are the holdout closest to deployment.
  • The test needs a pseudonymous organization key on every delivered record, so ask for it in the requirement.

What does a cross-company transfer test decide before a bigger purchase?

It decides what the next order should be: more records from the same companies, records from new companies, a narrower purchase, or nothing. The evidence that supports scaling is a gain over a fixed baseline that holds on supplier organizations the model never trained on, and on the buyer's own records, by a margin set before the pilot.

Consider an illustrative buyer that licensed a pilot tranche of invoice-exception records from six distributors to fine-tune an exception-routing model. The suppliers now offer ten times the volume. On a random test split of the six companies' records, the tuned model beats the current setup by 9 points. That number cannot say whether the model learned how exceptions get resolved or learned these six companies' reason codes, templates and staff habits, and only the first is worth buying ten times over.

The question this page answers is narrow on purpose. Whether data from ten companies beats data from one at an equal budget, and whether a model learned a workflow or one company's rules, are covered by the source-diversity protocol and the generalization test linked below. Here the pilot already exists, and the decision is the purchase that follows it.

Why a random record split overstates what more volume will buy

A random split puts records from the same company on both sides of the line, so the test rewards memorizing that company. Records from one organization share templates, codes, customers and people, and a model can score well by recognizing them.

Scikit-learn's documentation states the general rule: the independence assumption breaks when the data come in groups of dependent samples, and to learn whether a model trained on some groups generalizes to unseen groups, every validation sample must come from a group not represented at all in the paired training fold. For a data purchase, the group is the supplying company.

Medical imaging shows the cost of ignoring it. In a 2018 PLOS Medicine study, Zech and colleagues tested pneumonia-screening networks on chest radiographs from three hospital systems and found better internal than external performance in 3 of 5 natural comparisons; the networks could also identify which hospital system, and which department, an image came from. The WILDS benchmark, built from 10 datasets with real distribution shifts such as tumor images from different hospitals, found that standard training performs much worse on out-of-distribution test data than on in-distribution data for each dataset.

Even a fresh sample from the same source can cost accuracy. Recht and colleagues rebuilt ImageNet and CIFAR-10 test sets by repeating the original collection process and saw accuracy drops of about 11 to 14 percent on ImageNet and 3 to 15 percent on CIFAR-10. A different company with different software is a larger change than a recollection, so a buyer should expect some gap and measure its size.

Leave one supplier out, then score your own records last

Hold out one whole supplier organization at a time: train on the others, score on the one left out, and repeat until every supplier has been the holdout. Then score the buyer's own records once, at the end, as the organization that most resembles deployment.

Scikit-learn calls this scheme leave-one-group-out: each split holds out the samples of one group, with the group given by an identifier on every record. That identifier is the practical catch. The delivery must carry a stable, pseudonymous organization key on every record, or the test cannot be run at all.

Keep training size equal across folds, subsampling each fold to the smallest training set, so a fold does not look better only because it trained on more records. Run the same comparison with a random split of the training suppliers' records as well; the difference between the two gains is the transfer gap.

Three numbers summarize the run. The in-supplier gain is the lift on unseen records from familiar companies. The held-out gain is the average lift across organizations when each was left out. The transfer ratio divides the second by the first: near 1, the gain travels; near 0, it stayed with the companies it came from.

Treat the organization, not the record, as the unit of uncertainty. A 2024 paper on adding error bars to language model evaluations shows that when test questions come in related clusters, naive standard errors are inconsistent, and reports that the clustered adjustment in its real-world example was up to three times larger. With six suppliers, the interval for the held-out gain rests on six numbers, and it should look that wide.

The supplier holdout protocol to fix before training

Write eight elements down before the first training run: the decision, the holdout unit, the frozen conditions, the baseline, the primary metric, the secondary readings, the leakage checks and the decision rules. The defaults shown are the illustrative pilot's settings, editable assumptions rather than recommendations measured on real purchases. The protocol:

The supplier holdout protocol to fix before training
ElementWhat is fixed before trainingSetting in the illustrative pilot
DecisionScale volume, broaden to new suppliers, retarget sourcing, or stopWhether to buy ten times the volume from six distributors
Holdout unitWhole supplier organization, keyed by a pseudonymous organization ID on every recordSix organizations, one held out per fold
Frozen conditionsBase model, steps, tokens per fold equalized, prompts, harness and seedsThree seeds per fold
BaselineBest current alternative scored on the same foldsProduction setup with tuned prompts and retrieval
Primary metricMean held-out gain over baseline, with an interval across organizationsMinimum useful gain of 3 points
Secondary readingsIn-supplier gain, transfer ratio, worst organization, gain on the buyer's own recordsBuyer's records sealed until the fold verdict
Leakage checksShared customers, shared software templates and overlapping time windows across suppliersOverlapping entities logged and removed before splitting
Decision rulesSuccess, stop, inconclusive and void, as in the next sectionApplied to fold results before anyone reads the buyer's records

From transfer pattern to purchase: six results and the order each justifies

Each pattern in the results points to a different purchase, so map them before seeing any numbers. Success means the held-out gain clears the threshold with its interval above it and the buyer's own records agree; stop means no gain over a strong baseline anywhere; inconclusive means the interval straddles the threshold; void means a leak, a changed setup or a peek at the holdout during model selection.

Stop early only on a rule written in advance, for example when the first three folds all show a held-out gain below zero. Never stop early because the first folds look good. The six patterns and the order each supports:

From transfer pattern to purchase: six results and the order each justifies
Pattern in the pilotWhat it most likely meansNext purchaseWhat to ask suppliers
Held-out gain clears the threshold, interval above it, buyer's records agreeThe gain transfers to unseen companiesScale volume across the current supplier mixKeep one organization sealed for acceptance testing of each new tranche
Strong in-supplier gain, held-out gain near zeroThe model learned local codes and templatesBuy breadth: fewer records each from more organizationsOrganization keys and per-company record counts
Mean clears, but one or two organizations go negativeThe suppliers run different workflowsBuy for the segment that transfers; investigate the outliers firstHow the failing company's process and system differ
Held-out suppliers improve, buyer's own records do notThe supplier pool does not resemble deploymentRetarget sourcing toward companies like yoursCompany size, software and region of each supplier
No gain over a strong baseline in any foldData is not the bottleneck for this taskStop and revisit the task, prompts or retrievalNothing yet; return with a new requirement
Interval straddles the thresholdToo few organizations to tellAdd organizations to the pilot, not recordsWhether three or four more companies can join a pilot tranche

Six distributors, one inconclusive verdict: an illustrative run

In the illustrative distributor pilot, the transfer test turns a 9-point headline into an inconclusive verdict, and the right next step is more companies rather than more records. All figures below are invented to show the method; they are not results.

The in-supplier gain is 9.0 points. Held out one at a time, the six distributors show gains of 7, 6, 2, 5, -1 and 4 points, a mean of 3.8 and a transfer ratio of 0.43. A t-interval across the six organizations runs from about 0.8 to 6.9 points. The mean clears the 3-point threshold, but the interval straddles it, so the rule says inconclusive, and distributor E, the only negative fold, gets a separate look at its workflow and system.

The buyer's own exception queue stays sealed. Scoring it now would spend the one holdout that most resembles deployment on a verdict that is not yet decided. The order that follows is a pilot tranche from three or four more distributors, not ten times the volume from these six.

A dot chart of held-out gains for six illustrative distributors, A to F, against a 9-point in-supplier line and a 3-point threshold, with the mean of 3.8 and its interval of 0.8 to 6.9 straddling the threshold and distributor E below zero.
Illustrative: a 9-point pilot headline shrinks to 3.8 points on held-out suppliers. Values are examples, not results.

Could the transfer gain be an artifact of the setup?

It can be. A transfer gain can shrink under a stronger baseline, a stricter holdout or more trials, so test each before reading the gain as a property of the data.

At a glance

Could the transfer gain be an artifact of the setup

  • A weak baseline: in the DomainBed study, Gulrajani and Lopez-Paz found that with careful implementation, no domain generalization method beat plain empirical risk minimization by more than one point on average under any model selection criterion. Compare the candidate data with a well-tuned current setup, not an untuned model.
  • Model selection that peeks: the same paper argues that a method without a model selection strategy is incomplete. Choose checkpoints and settings on the training suppliers only, never on the held-out one.
  • Leaks across suppliers: two distributors can share customers, run the same ERP with identical reason codes, or cover the same weeks. Resolve shared entities and split by connected groups before the folds are built.
  • Too few trials: Anthropic's guidance treats each attempt at a task as a separate trial and measures how consistently an agent passes across several. Run several seeds per fold and check that the spread across seeds is small next to the spread across organizations.
  • Fold size: a small held-out company yields a noisy estimate. Report each organization's gain with its record count, and weight the primary metric by organization rather than by record.

Limits of cross-company dataset generalization evidence

A supplier holdout measures transfer among the companies that agreed to license, which is not a random sample of the market. Companies that license may keep cleaner records or run more standard software than those that do not, so a passing result says the gain travels within that pool and suggests, without proving, that it travels further.

The result also has a shelf life. A model upgrade can close the gap between familiar and unfamiliar companies, or open a new one, so the test belongs in each purchase cycle rather than once. And it prices nothing: a transfer ratio says where value generalizes, not what a tranche is worth.

The next evidence to collect is more organizations in the pilot, the buyer's own records scored once the fold verdict is success, and a later time window to check that the gain survives the passage of time as well as the change of company.

Asking for organization keys in the data request

Put the holdout into the requirement: ask for the number of distinct organizations, a pseudonymous organization key on every record, per-organization record counts and date ranges, the system each company uses, and permission to keep one organization sealed for acceptance testing.

SourceX sources datasets for buyers from U.S. businesses on request rather than holding them in stock, with scope, volume, language, format, timing and licensing needs captured in the request form. Each supplying company approves the scope and terms in writing, with its signature, before handing over data, so a holdout use belongs in the scope from the start. Whether enough comparable companies can be found for a given task is assessed case by case; nothing here implies available stock or a particular result.

Frequently asked questions

What is cross-company dataset generalization?

It is whether a model trained on records from some companies performs well at companies it never saw. You measure it by holding out whole organizations rather than random records, because records from one company share codes, templates and customers that a model can memorize.

How many supplier organizations does a transfer test need?

Each held-out organization contributes one number to the interval, so three organizations give a very wide interval. As a rule of thumb for planning, aim for at least five or six, and when the result is inconclusive add organizations rather than records.

Why not use a random train and test split of all the records?

A random split places records from the same company on both sides, so the test rewards learning that company. Scikit-learn's documentation notes that measuring generalization to unseen groups requires validation samples from groups absent from training.

What if one supplier's records make the model worse?

Do not average it away. Check for leakage and setup errors, then compare that company's process and software with the others. A real outlier usually means the purchase should target the segment that transfers.

Should the buyer's own data be part of the pilot?

Yes, as a final sealed holdout scored once after the supplier folds reach a success verdict. Never use it to choose settings, or it stops being an independent test of deployment.

Does this replace a one-company versus ten-company study?

No. That study asks whether source diversity adds value at a fixed budget. A supplier holdout decides the next purchase after a pilot, using the companies already in it.

Sources

  • Scikit-learn's cross-validation guide says the independence assumption breaks with groups of dependent samples, that testing generalization to unseen groups requires validation samples from groups absent from training, and describes leave-one-group-out splitting. Source
  • Zech et al. (PLOS Medicine, 2018) found pneumonia-screening CNNs had better internal than external performance in 3 of 5 natural comparisons across three hospital systems, and that the networks could identify the hospital system and department of an image. Source
  • WILDS curates 10 datasets with real-world distribution shifts, such as across hospitals, and finds standard training performs much worse out of distribution than in distribution on each. Source
  • Recht et al. (2019) rebuilt CIFAR-10 and ImageNet test sets by replicating the original process and observed accuracy drops of 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet. Source
  • Gulrajani and Lopez-Paz (DomainBed) found that with careful implementation, no domain generalization method improves on ERM's average performance by more than one point under any model selection criterion, and that methods without a model selection strategy are incomplete. Source
  • Adding Error Bars to Evals (2024) shows that clustered test questions make naive standard errors inconsistent, recommends clustered standard errors, and reports adjustments up to three times in a real example. Source
  • Anthropic's guide to agent evals defines a trial as each attempt at a task and discusses repeated trials and consistency metrics. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies