AI uses for records
Cross-company dataset generalization: test held-out suppliers before scaling
By SourceX Editorial · Updated
9 sections
Short answer
Before scaling a data purchase, test cross-company dataset generalization by holding out whole supplier organizations: train on all but one, score on the one left out, and repeat for each. Compare that held-out gain with the gain on familiar suppliers and on your own records, with intervals that count organizations, not records. Scale only if the gain survives unseen companies; otherwise buy breadth, not volume.
Evaluation split
Cross company dataset
Key takeaways
- A random record split rewards a model for memorizing the supplying companies; hold out whole organizations instead.
- The transfer ratio, held-out gain divided by in-supplier gain, shows how much of a pilot's headline travels to unseen companies.
- Organizations are the unit of uncertainty: six suppliers give six numbers, and the interval should look that wide.
- When the interval straddles the threshold, the next order is more companies in the pilot, not more records from the same ones.
- Keep the buyer's own records sealed until the supplier folds pass; they are the holdout closest to deployment.
- The test needs a pseudonymous organization key on every delivered record, so ask for it in the requirement.
What does a cross-company transfer test decide before a bigger purchase?
It decides what the next order should be: more records from the same companies, records from new companies, a narrower purchase, or nothing. The evidence that supports scaling is a gain over a fixed baseline that holds on supplier organizations the model never trained on, and on the buyer's own records, by a margin set before the pilot.
Consider an illustrative buyer that licensed a pilot tranche of invoice-exception records from six distributors to fine-tune an exception-routing model. The suppliers now offer ten times the volume. On a random test split of the six companies' records, the tuned model beats the current setup by 9 points. That number cannot say whether the model learned how exceptions get resolved or learned these six companies' reason codes, templates and staff habits, and only the first is worth buying ten times over.
The question this page answers is narrow on purpose. Whether data from ten companies beats data from one at an equal budget, and whether a model learned a workflow or one company's rules, are covered by the source-diversity protocol and the generalization test linked below. Here the pilot already exists, and the decision is the purchase that follows it.
Why a random record split overstates what more volume will buy
A random split puts records from the same company on both sides of the line, so the test rewards memorizing that company. Records from one organization share templates, codes, customers and people, and a model can score well by recognizing them.
Scikit-learn's documentation states the general rule: the independence assumption breaks when the data come in groups of dependent samples, and to learn whether a model trained on some groups generalizes to unseen groups, every validation sample must come from a group not represented at all in the paired training fold. For a data purchase, the group is the supplying company.
Medical imaging shows the cost of ignoring it. In a 2018 PLOS Medicine study, Zech and colleagues tested pneumonia-screening networks on chest radiographs from three hospital systems and found better internal than external performance in 3 of 5 natural comparisons; the networks could also identify which hospital system, and which department, an image came from. The WILDS benchmark, built from 10 datasets with real distribution shifts such as tumor images from different hospitals, found that standard training performs much worse on out-of-distribution test data than on in-distribution data for each dataset.
Even a fresh sample from the same source can cost accuracy. Recht and colleagues rebuilt ImageNet and CIFAR-10 test sets by repeating the original collection process and saw accuracy drops of about 11 to 14 percent on ImageNet and 3 to 15 percent on CIFAR-10. A different company with different software is a larger change than a recollection, so a buyer should expect some gap and measure its size.
Leave one supplier out, then score your own records last
Hold out one whole supplier organization at a time: train on the others, score on the one left out, and repeat until every supplier has been the holdout. Then score the buyer's own records once, at the end, as the organization that most resembles deployment.
Scikit-learn calls this scheme leave-one-group-out: each split holds out the samples of one group, with the group given by an identifier on every record. That identifier is the practical catch. The delivery must carry a stable, pseudonymous organization key on every record, or the test cannot be run at all.
Keep training size equal across folds, subsampling each fold to the smallest training set, so a fold does not look better only because it trained on more records. Run the same comparison with a random split of the training suppliers' records as well; the difference between the two gains is the transfer gap.
Three numbers summarize the run. The in-supplier gain is the lift on unseen records from familiar companies. The held-out gain is the average lift across organizations when each was left out. The transfer ratio divides the second by the first: near 1, the gain travels; near 0, it stayed with the companies it came from.
Treat the organization, not the record, as the unit of uncertainty. A 2024 paper on adding error bars to language model evaluations shows that when test questions come in related clusters, naive standard errors are inconsistent, and reports that the clustered adjustment in its real-world example was up to three times larger. With six suppliers, the interval for the held-out gain rests on six numbers, and it should look that wide.
The supplier holdout protocol to fix before training
Write eight elements down before the first training run: the decision, the holdout unit, the frozen conditions, the baseline, the primary metric, the secondary readings, the leakage checks and the decision rules. The defaults shown are the illustrative pilot's settings, editable assumptions rather than recommendations measured on real purchases. The protocol:
| Element | What is fixed before training | Setting in the illustrative pilot |
|---|---|---|
| Decision | Scale volume, broaden to new suppliers, retarget sourcing, or stop | Whether to buy ten times the volume from six distributors |
| Holdout unit | Whole supplier organization, keyed by a pseudonymous organization ID on every record | Six organizations, one held out per fold |
| Frozen conditions | Base model, steps, tokens per fold equalized, prompts, harness and seeds | Three seeds per fold |
| Baseline | Best current alternative scored on the same folds | Production setup with tuned prompts and retrieval |
| Primary metric | Mean held-out gain over baseline, with an interval across organizations | Minimum useful gain of 3 points |
| Secondary readings | In-supplier gain, transfer ratio, worst organization, gain on the buyer's own records | Buyer's records sealed until the fold verdict |
| Leakage checks | Shared customers, shared software templates and overlapping time windows across suppliers | Overlapping entities logged and removed before splitting |
| Decision rules | Success, stop, inconclusive and void, as in the next section | Applied to fold results before anyone reads the buyer's records |
From transfer pattern to purchase: six results and the order each justifies
Each pattern in the results points to a different purchase, so map them before seeing any numbers. Success means the held-out gain clears the threshold with its interval above it and the buyer's own records agree; stop means no gain over a strong baseline anywhere; inconclusive means the interval straddles the threshold; void means a leak, a changed setup or a peek at the holdout during model selection.
Stop early only on a rule written in advance, for example when the first three folds all show a held-out gain below zero. Never stop early because the first folds look good. The six patterns and the order each supports:
| Pattern in the pilot | What it most likely means | Next purchase | What to ask suppliers |
|---|---|---|---|
| Held-out gain clears the threshold, interval above it, buyer's records agree | The gain transfers to unseen companies | Scale volume across the current supplier mix | Keep one organization sealed for acceptance testing of each new tranche |
| Strong in-supplier gain, held-out gain near zero | The model learned local codes and templates | Buy breadth: fewer records each from more organizations | Organization keys and per-company record counts |
| Mean clears, but one or two organizations go negative | The suppliers run different workflows | Buy for the segment that transfers; investigate the outliers first | How the failing company's process and system differ |
| Held-out suppliers improve, buyer's own records do not | The supplier pool does not resemble deployment | Retarget sourcing toward companies like yours | Company size, software and region of each supplier |
| No gain over a strong baseline in any fold | Data is not the bottleneck for this task | Stop and revisit the task, prompts or retrieval | Nothing yet; return with a new requirement |
| Interval straddles the threshold | Too few organizations to tell | Add organizations to the pilot, not records | Whether three or four more companies can join a pilot tranche |
Six distributors, one inconclusive verdict: an illustrative run
In the illustrative distributor pilot, the transfer test turns a 9-point headline into an inconclusive verdict, and the right next step is more companies rather than more records. All figures below are invented to show the method; they are not results.
The in-supplier gain is 9.0 points. Held out one at a time, the six distributors show gains of 7, 6, 2, 5, -1 and 4 points, a mean of 3.8 and a transfer ratio of 0.43. A t-interval across the six organizations runs from about 0.8 to 6.9 points. The mean clears the 3-point threshold, but the interval straddles it, so the rule says inconclusive, and distributor E, the only negative fold, gets a separate look at its workflow and system.
The buyer's own exception queue stays sealed. Scoring it now would spend the one holdout that most resembles deployment on a verdict that is not yet decided. The order that follows is a pilot tranche from three or four more distributors, not ten times the volume from these six.
Could the transfer gain be an artifact of the setup?
It can be. A transfer gain can shrink under a stronger baseline, a stricter holdout or more trials, so test each before reading the gain as a property of the data.
At a glance
Could the transfer gain be an artifact of the setup
- A weak baseline: in the DomainBed study, Gulrajani and Lopez-Paz found that with careful implementation, no domain generalization method beat plain empirical risk minimization by more than one point on average under any model selection criterion. Compare the candidate data with a well-tuned current setup, not an untuned model.
- Model selection that peeks: the same paper argues that a method without a model selection strategy is incomplete. Choose checkpoints and settings on the training suppliers only, never on the held-out one.
- Leaks across suppliers: two distributors can share customers, run the same ERP with identical reason codes, or cover the same weeks. Resolve shared entities and split by connected groups before the folds are built.
- Too few trials: Anthropic's guidance treats each attempt at a task as a separate trial and measures how consistently an agent passes across several. Run several seeds per fold and check that the spread across seeds is small next to the spread across organizations.
- Fold size: a small held-out company yields a noisy estimate. Report each organization's gain with its record count, and weight the primary metric by organization rather than by record.
Limits of cross-company dataset generalization evidence
A supplier holdout measures transfer among the companies that agreed to license, which is not a random sample of the market. Companies that license may keep cleaner records or run more standard software than those that do not, so a passing result says the gain travels within that pool and suggests, without proving, that it travels further.
The result also has a shelf life. A model upgrade can close the gap between familiar and unfamiliar companies, or open a new one, so the test belongs in each purchase cycle rather than once. And it prices nothing: a transfer ratio says where value generalizes, not what a tranche is worth.
The next evidence to collect is more organizations in the pilot, the buyer's own records scored once the fold verdict is success, and a later time window to check that the gain survives the passage of time as well as the change of company.
Asking for organization keys in the data request
Put the holdout into the requirement: ask for the number of distinct organizations, a pseudonymous organization key on every record, per-organization record counts and date ranges, the system each company uses, and permission to keep one organization sealed for acceptance testing.
SourceX sources datasets for buyers from U.S. businesses on request rather than holding them in stock, with scope, volume, language, format, timing and licensing needs captured in the request form. Each supplying company approves the scope and terms in writing, with its signature, before handing over data, so a holdout use belongs in the scope from the start. Whether enough comparable companies can be found for a given task is assessed case by case; nothing here implies available stock or a particular result.
Frequently asked questions
What is cross-company dataset generalization?
It is whether a model trained on records from some companies performs well at companies it never saw. You measure it by holding out whole organizations rather than random records, because records from one company share codes, templates and customers that a model can memorize.
How many supplier organizations does a transfer test need?
Each held-out organization contributes one number to the interval, so three organizations give a very wide interval. As a rule of thumb for planning, aim for at least five or six, and when the result is inconclusive add organizations rather than records.
Why not use a random train and test split of all the records?
A random split places records from the same company on both sides, so the test rewards learning that company. Scikit-learn's documentation notes that measuring generalization to unseen groups requires validation samples from groups absent from training.
What if one supplier's records make the model worse?
Do not average it away. Check for leakage and setup errors, then compare that company's process and software with the others. A real outlier usually means the purchase should target the segment that transfers.
Should the buyer's own data be part of the pilot?
Yes, as a final sealed holdout scored once after the supplier folds reach a success verdict. Never use it to choose settings, or it stops being an independent test of deployment.
Does this replace a one-company versus ten-company study?
No. That study asks whether source diversity adds value at a fixed budget. A supplier holdout decides the next purchase after a pilot, using the companies already in it.
Sources
- Scikit-learn's cross-validation guide says the independence assumption breaks with groups of dependent samples, that testing generalization to unseen groups requires validation samples from groups absent from training, and describes leave-one-group-out splitting. Source
- Zech et al. (PLOS Medicine, 2018) found pneumonia-screening CNNs had better internal than external performance in 3 of 5 natural comparisons across three hospital systems, and that the networks could identify the hospital system and department of an image. Source
- WILDS curates 10 datasets with real-world distribution shifts, such as across hospitals, and finds standard training performs much worse out of distribution than in distribution on each. Source
- Recht et al. (2019) rebuilt CIFAR-10 and ImageNet test sets by replicating the original process and observed accuracy drops of 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet. Source
- Gulrajani and Lopez-Paz (DomainBed) found that with careful implementation, no domain generalization method improves on ERM's average performance by more than one point under any model selection criterion, and that methods without a model selection strategy are incomplete. Source
- Adding Error Bars to Evals (2024) shows that clustered test questions make naive standard errors inconsistent, recommends clustered standard errors, and reports adjustments up to three times in a real example. Source
- Anthropic's guide to agent evals defines a trial as each attempt at a task and discusses repeated trials and consistency metrics. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.