AI uses for records
How to measure whether ten source companies beat one: a source diversity experiment protocol
By SourceX Editorial · Updated
10 sections
Short answer
This is a protocol, not a result. To measure whether data from ten companies is worth more than data from one, train on the same total number of cases drawn from one, two, five or ten companies, repeat with different company draws and seeds, and score every model on companies held out from all training arms. Compare per-company performance with intervals that resample companies, and pre-register the question, arms, metric and analysis.
Agent run
Dataset source diversity
Key takeaways
- Hold the total case budget fixed across arms, or source count and case count are confounded.
- Evaluate on whole companies that no arm ever trained on; a random case split rewards memorizing familiar sources.
- Repeat each arm over different company subsets and seeds, so the result is about how many companies and not which ones.
- Treat companies as the unit of uncertainty: intervals should resample held-out companies as well as cases.
- Pre-register arms, metrics and the analysis before anyone sees a score; this page reports no results.
How do you test the incremental utility of data from one company versus ten?
Run a controlled comparison at an equal case budget: build training sets of the same size drawn from one company, from two, from five and from ten, train the same model recipe on each, and measure performance on companies that none of the training sets touched. Repeat each arm with different company draws and training seeds, and report the per-company result with intervals that treat companies as the sampling unit.
That is the core of a dataset source diversity utility experiment. Everything else in the protocol exists to stop a different explanation from posing as a diversity effect: more cases, easier companies, leaked duplicates, a lucky seed or a metric chosen after the fact.
This page publishes the method only. It reports no results, and no figure on it should be read as a finding. Arm sizes and the work order setting below are illustrative design values that a team replaces with its own.
Why a pooled-data gain is easy to misread
A model trained on many sources can look better simply because it was tested on data that resembles its training sources. The WILDS benchmark, which collects real distribution shifts such as tumor images across hospitals, reports that standard training gives substantially lower out-of-distribution than in-distribution performance on each of its datasets. A test drawn from the same companies as training hides exactly the gap a buyer cares about.
Pooling can also teach the source rather than the task. Zech and colleagues found that a pneumonia model trained on pooled radiographs from two hospital systems did not reproduce its performance at a third, and that it relied on features tied to the hospital and department, which also predicted the outcome in the pooled data. Company-level signals, such as one firm's ticket templates or coding habits, can play the same role in business records.
Volume confounds everything. Kaplan and colleagues report that language model test loss follows a power law in dataset size, so adding companies usually adds cases too, and any gain could come from the extra cases alone. The comparison has to hold the case count fixed and model case count separately, on a log scale.
The controlled case count and source diversity study
The table is the study design. Each row is a design element, what to specify, and the false conclusion it prevents. Fill every row before collecting scores; a row left open is where an analyst's choices will creep in later.
| Design element | Specification | False conclusion it prevents |
|---|---|---|
| Research question | At a fixed case budget, does raising the number of source companies improve performance on unseen companies, and by how much per doubling? | A vague better-data claim that cannot be falsified |
| Population | A pool of companies with the same record type, schema and label definition, with eligibility rules written down | Comparing sources that do not perform the same task |
| Sampling | For each arm, many random company subsets of the arm's size; within each company, random cases up to its share of the budget | Crediting diversity for one unusually good company |
| Variables | Source count and total cases manipulated; company identity, company size, period and system recorded as covariates | Mixing up how many sources with which sources |
| Comparison | Every arm against the one-company arm at the same budget, plus a one-company arm at several times the budget | Missing whether more cases from one source would do as well |
| Holdout | Whole companies excluded from all arms, plus held-out cases from training companies | Scoring familiarity instead of generalization |
| Metrics | Primary task metric averaged per held-out company, worst held-out company, in-source minus out-of-source gap | A pooled average dominated by one large company |
| Uncertainty | Intervals resampling held-out companies and cases, several training seeds per subset, power analysis before the run | A difference that is one seed's noise |
| Limits | Stated in advance: task, model family, budget range and company population the result can speak for | Extrapolating to other tasks or model sizes |
Arms at an equal case budget
Cross source count with total case count so each effect can be read separately. The grid below is illustrative: the budgets and company counts are placeholders to show the structure, and a team sets them from its own pool and compute.
Keep cases per company balanced within an arm where possible. If some companies have too few cases for their share, record the shortfall rather than topping up from other companies, because topping up quietly changes the source mix.
Report every arm the same way: the companies drawn for each subset, the cases used per company, the seeds, the score on each held-out company and the in-source score. A results table that lists only the best subset per arm, or averages away the worst held-out company, hides the variation the design was built to measure.
| Arm | Source companies | Cases per company | Total cases | Purpose |
|---|---|---|---|---|
| A1 | 1 | Full budget | Budget B | Single-source baseline |
| A2 | 2 | B divided by 2 | Budget B | First step in diversity |
| A5 | 5 | B divided by 5 | Budget B | Middle of the curve |
| A10 | 10 | B divided by 10 | Budget B | Full diversity at equal budget |
| V1 | 1 | Several times B | Several times B | Does volume from one source match diversity? |
| Grid repeat | 1, 2, 5 and 10 | Scaled | Smaller and larger budgets | Whether the diversity effect changes with budget |
Which companies must the model never see?
Set aside whole companies before drawing any training arm, and never let them into training, hyperparameter tuning or early stopping. These held-out companies are the test of what a buyer actually wants to know: how the model performs on a business it has never met. Keep a second, in-source test set of held-out cases from training companies to measure the familiarity gap.
Model selection needs the same discipline. Gulrajani and Lopez-Paz argue that domain generalization methods without a stated model selection strategy are incomplete, and they found that plain empirical risk minimization matched the specialized methods under fair selection. Choose checkpoints and settings using validation data from training companies only, use the same tuning budget for every arm, and write the rule down before running.
Remove near-duplicates across companies before splitting. Franchisees, shared vendors and template libraries can put almost identical records in two companies, and one copy in training with the other in the held-out set inflates the out-of-source score.
Choose the held-out companies by a recorded random draw, stratified if the pool mixes very different company sizes or systems, and publish the draw procedure with the plan. Hand-picking held-out companies, even with good intentions, lets an analyst choose an easy or a hard test without meaning to.
Metrics and uncertainty: counting companies, not just cases
The unit of generalization is the company, so the primary metric is the task score computed per held-out company and then averaged, with the worst held-out company reported alongside. A pooled score over all held-out cases lets one large company dominate.
Several sources of variance stack up here. Bouthillier and colleagues show that data sampling, parameter initialization and hyperparameter choices each noticeably move benchmark results, so each arm needs several company subsets and several seeds per subset. Build intervals with a resampling scheme that draws held-out companies first and cases within them second; an interval over cases alone will be too narrow.
Check power before collecting. Card and colleagues argue that statistical power has been largely neglected in NLP experiments and recommend power analysis as standard practice. Here power depends mostly on the number of held-out companies, not the number of test cases, so a study with three held-out companies may not be able to detect any plausible effect however many cases each has. Decide the smallest effect worth detecting, and size the held-out pool to it.
For the main analysis, fit the held-out score as a function of the logarithm of source count and the logarithm of total cases, with the held-out company and the training subset as grouping factors. The coefficient on source count, at fixed cases, is the estimate the study exists to produce; report it with its interval and the per-doubling change it implies, and show the raw per-arm scores beside it so readers can see the model is not hiding a nonlinear pattern.
What alternative explanations could produce an apparent gain?
List the rivals before the run and attach a check to each; an apparent diversity effect counts only if it survives all of them.
At a glance
What alternative explanations could produce an apparent gain
- More information per case: diverse arms may contain longer or richer records. Check token counts and field completeness by arm, and match them or adjust.
- Label definition drift: companies may code the same outcome differently. Audit a sample of labels per company against one written definition before including it.
- Easier companies in the diverse arms: random subsets can tilt toward cleaner data. Record per-company difficulty from the one-company arms and adjust for it.
- Leakage across companies: shared templates, vendors or customers. Deduplicate across the whole pool before splitting.
- Period effects: companies may cover different years. Record each company's date range and compare arms within matched periods where possible.
- Tuning advantage: one arm may have received more tuning attention. Fix the tuning budget and rule per arm in the plan.
- Metric choice after the fact: the arm that wins can change with the metric. Name the primary metric in the pre-registration and report the rest as secondary.
Pre-registration checklist
Write the plan and time-stamp it before any arm is trained. Nosek and colleagues describe preregistration as the way to keep prediction separate from postdiction: hypotheses and analysis are fixed before the data can suggest them. For a data utility study, the checklist below is the minimum.
Checklist · 9 items
- The question, with the direction and smallest effect of interest for the source-count effect.
- Company eligibility rules, the pool, and the held-out companies chosen by a recorded random draw.
- Arms, budgets, number of company subsets per arm and seeds per subset.
- Model recipe, tuning budget and model selection rule, identical across arms.
- Primary metric, secondary metrics, per-company averaging and the worst-company report.
- The resampling scheme for intervals and the power analysis behind the held-out pool size.
- Deduplication method, label audit procedure and exclusion rules, with the count excluded at each step.
- What will be reported whatever the outcome, including null and negative results.
- The limits: task, model family, budget range and population the conclusion covers.
0 of 9 ticked. Tick each item as you go.
Applying the protocol to field-service work orders (illustrative)
As an illustration of the design only, imagine a buyer evaluating a model that predicts whether a field-service work order will need a second visit. A pool of work orders from sixteen companies on comparable systems is split by a recorded draw into twelve training-eligible companies and four held-out companies. Arms draw one, two, five and ten companies from the twelve, each subset repeated several times, all at the same case budget, plus a one-company arm at several times the budget.
Before training, the team checks that all sixteen companies define a second visit the same way, removes duplicate orders shared through a common subcontractor, and records each company's date range. The primary metric is per-company performance on the four held-out companies; the plan also reports the worst of the four.
Whatever the outcome, the conclusion is limited to second-visit prediction, that model family, that budget range and companies like those in the pool. No result is implied here; the example shows where each design choice lands in a real record type.
Assembling records from many companies for the study
The hardest input is a pool of comparable companies with consistent record types, enough history each, and permission to use the records for training and evaluation. Rights and consent vary by company, so each source needs its own review, and held-out companies need the same terms as training ones.
For a multi-company pool, SourceX can search U.S. businesses once a buyer files a request naming the record type, volume per company, language, format, timing and licensing terms. As a general guide, buyers tend to look for years of consistent history and tens of thousands of records or more in one system per source, though smaller sets can qualify when the work is rare, and each case is assessed on its own.
Frequently asked questions
Why not just compare a ten-company dataset with a one-company dataset of whatever size each has?
Because the larger set would usually win on volume alone, and the comparison could not separate the two effects. Holding the total case budget fixed across arms isolates the source-count effect, and a separate one-company arm at a larger budget shows whether more cases from one source would do as well.
Can a random train and test split measure source diversity?
No. A random split puts cases from the same companies in both training and test, so it rewards familiarity with known sources. Hold out whole companies that no arm trains on, and keep a within-company test set only to measure the familiarity gap.
What alternative explanation could account for an apparent improvement from more sources?
The diverse arms may have richer records, easier companies, cross-company duplicates or more tuning attention, or the metric may have been chosen after seeing results. Each rival needs a check written into the plan, such as matched token counts, per-company difficulty, deduplication and a fixed tuning budget.
How many held-out companies does the experiment need?
Enough to detect the smallest effect you care about, which a power analysis decides. Power depends mainly on the number of held-out companies rather than test cases, so a design with very few held-out companies can fail to detect a real effect regardless of test set size.
Does this page report whether diversity helps?
No. It is a protocol that sets out the question, design, metrics, uncertainty and limits. Any result depends on the task, model, budget and company pool, and should be reported against a pre-registered plan.
Sources
- WILDS collects real-world distribution shifts, such as across hospitals, and reports that standard training gives substantially lower out-of-distribution than in-distribution performance on each dataset. Source
- A pneumonia model trained on pooled data from two hospital systems did not reproduce its performance at a third system and relied on hospital-related image features. Source
- Language model test loss follows power laws in model size, dataset size and compute. Source
- Domain generalization methods without a model selection strategy should be regarded as incomplete, and empirical risk minimization matched specialized methods under fair selection. Source
- Variance from data sampling, parameter initialization and hyperparameter choice noticeably affects machine learning benchmark comparisons. Source
- Statistical power has been largely neglected in NLP experiments, and the authors recommend power analysis as standard practice. Source
- Preregistration keeps prediction separate from postdiction by fixing hypotheses and analysis before data are observed. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.