AI uses for records
Supplier clustered dataset sampling: count suppliers, not just records
By SourceX Editorial · Updated
9 sections
Short answer
In supplier clustered dataset sampling, the contributing company, not the record, sets how much a dataset can prove. Records from one supplier share templates, staff, customers and policies, so 48,000 records from six companies can behave like a few hundred independent cases when records within a supplier are even mildly alike. Estimate the design effect, cluster evaluation intervals by supplier, cap any one supplier's share and test on held-out suppliers.
Agent run
Supplier clustered dataset
Key takeaways
- Records from one supplier are correlated, so the supplier, not the record, is the unit that limits evidence.
- With large clusters, effective sample size approaches the number of suppliers divided by the intra-class correlation, so more records from the same suppliers barely move it.
- Cluster evaluation standard errors by supplier; on the same data, Anthropic's eval statistics paper found clustered errors can exceed naive ones by more than three times.
- Hold whole suppliers out to test whether a model generalizes to new companies, as group splitters and hospital-level benchmark splits do.
- Write supplier counts, a per-supplier cap, carried supplier codes and a held-out-supplier test into the data request.
Why does one supplier's share of a dataset change what it can prove?
Because records from the same company are alike in ways that have nothing to do with the task. They share templates, staff, customers, policies and software, so each extra record from a supplier you already have adds less new information than a record from a new one. The supplier, not the record, limits how far results generalize.
Supplier clustered dataset sampling means planning, measuring and reporting a dataset with that structure in view. It applies to any dataset assembled from several companies' business records, which describes most licensed operational data: a support corpus pooled from five software vendors, claims files from three insurers, work orders from a dozen field-service firms.
The decision this guide supports is concrete. Before accepting a coverage or accuracy claim, or before paying for more records, decide whether the evidence is limited by records or by suppliers, because the answer changes what you ask for next.
Where supplier clustering comes from in business records
Clustering comes from each supplier's process, not from duplication. Even when every record is a distinct case, one company's records carry the fingerprints of how that company works, and a model can learn the fingerprints instead of the task.
At a glance
Where supplier clustering comes from in business records
Machine learning benchmarks build this structure in on purpose. WILDS, a benchmark of real-world distribution shifts, includes a tumor identification task in which each image patch's domain is the hospital that supplied it, one of five, with training and test patches drawn from separate hospitals. Its documentation describes metadata recording domain identity, such as which hospital a patient's data came from, so results can be grouped and compared across sources.
Clustering also nests. Records sit inside cases, cases inside agents or teams, teams inside sites and sites inside suppliers, and similarity can arise at every level. The supplier is usually the top level and the one a buyer controls when sourcing. Within-supplier similarity usually comes from five places:
- Templates and required fields: one company's ticket, claim or work order form shapes what every record says and leaves out.
- People: a handful of agents, adjusters or technicians write most of a small supplier's records, each with stable habits.
- Customers and regions: a supplier's case mix mirrors its own customers, products and geography.
- Policies: refund rules, escalation thresholds and approval limits set outcomes before any judgment is recorded.
- Systems: the software and its configuration decide timestamps, codes and what gets captured at all.
How many independent cases does a clustered dataset really hold?
Divide the record count by the design effect. For equal-sized clusters the design effect is approximately 1 + (m - 1) × ICC, where m is the number of records per supplier and ICC, the intra-class correlation, measures how much more alike two records from one supplier are than two records from different suppliers.
Survey statisticians handle this arithmetic routinely. The United Nations handbook on designing household survey samples defines the design effect as the ratio of an estimate's true sampling variance to the variance a simple random sample of the same size would give, and explains that the clustering part depends on two things: how many units are taken per cluster and the intra-class correlation. It also warns against adopting a design effect from another survey, which applies just as much to borrowing one from another dataset.
The arithmetic has an uncomfortable ceiling. As records per supplier grow, the effective sample size approaches the number of suppliers divided by the ICC, however many records each supplier adds. In the illustration below, 48,000 records from six suppliers with an ICC of 0.02 behave like about 300 independent cases; 6,000 records from the same six suppliers behave like about 286, so the extra 42,000 records buy only about a dozen. Spread across 30 suppliers, the same 48,000 records behave like about 1,455.
The ICC values are assumptions chosen to show the mechanism. Measure your own on the metric that matters, because the ICC for an outcome label can differ sharply from the ICC for note length or an error rate. Unequal supplier sizes generally make matters worse than the equal-size formula suggests, so treat its result as an upper bound on the evidence.
| Design (Illustrative) | Records per supplier | ICC | Design effect | Effective cases |
|---|---|---|---|---|
| 6 suppliers, 48,000 records | 8,000 | 0.01 | 81.0 | About 593 |
| 6 suppliers, 48,000 records | 8,000 | 0.02 | 161.0 | About 298 |
| 6 suppliers, 48,000 records | 8,000 | 0.05 | 401.0 | About 120 |
| 6 suppliers, 6,000 records | 1,000 | 0.02 | 21.0 | About 286 |
| 30 suppliers, 48,000 records | 1,600 | 0.02 | 33.0 | About 1,455 |
| 60 suppliers, 48,000 records | 800 | 0.02 | 17.0 | About 2,827 |
The supplier sampling worksheet
Fill the worksheet before buying, splitting or reporting anything. Each field can be computed from a manifest that carries a supplier code on every record, and each drives one decision.
Keep the supplier code, and the site, team and case codes beneath it, through every stage of preparation; a pseudonymous code is enough. Once the codes are stripped for privacy or convenience, none of these fields can be recomputed, and every later claim falls back to a raw record count.
The values in the third column belong to one illustrative dataset of 48,000 records from six suppliers whose shares run from 44 percent down to 4 percent.
| Field | How to fill it | Illustrative value | Decision it drives |
|---|---|---|---|
| Target population | Name the companies, workflows and period a claim must cover | U.S. home-services firms, dispatch to invoice, 2019 to 2026 | Which suppliers are in scope at all |
| Supplier count | Count distinct supplier codes in the manifest | 6 | Whether any claim beyond these firms is supportable |
| Records per supplier | Report minimum, median and maximum | 1,920 / 5,760 / 21,120 | Where caps and weights are needed |
| Largest supplier share | Largest supplier's records divided by all records | 44% | Whether one company dominates every metric |
| Effective number of suppliers | One divided by the sum of squared supplier shares | 3.7 | How many companies the data really represents |
| ICC for the key metric | Estimate from a pilot sample scored across every supplier | 0.02, from a 600-record pilot | Design effect and the supplier count needed |
| Effective sample size | Records divided by the design effect | About 300, using a mean of 8,000 per supplier | Whether more records or more suppliers come next |
| Held-out suppliers | Suppliers kept out of training, tuning and curation | 1 of 6, rotated | Whether a generalization claim can be tested |
| Per-supplier cap | Maximum share any supplier may contribute to a set | 25% | Downsampling or new sourcing |
Evaluation intervals: cluster by supplier, then hold suppliers out
Report evaluation results with standard errors clustered on the supplier, and test generalization on suppliers the model never saw. The first keeps the interval honest about how much evidence exists; the second answers whether the model works for the next company, which is usually what a buyer is paying for.
Anthropic's guidance on statistics for model evaluations makes the same move for benchmarks: it recommends clustering standard errors on the unit of randomization when questions are not independent, such as several questions about one passage. The accompanying paper, Adding Error Bars to Evals, reports that on the same data, clustered standard errors can be more than three times the naive ones. Because a standard error scales with the square root of the design effect, a threefold widening corresponds to a design effect of about nine.
Then split by supplier. scikit-learn's documentation explains that grouped data breaks the independence assumption, that GroupKFold keeps any one group from appearing on both the training and test side of a split, and that LeaveOneGroupOut holds out each group in turn. With the supplier code as the group, leave-one-supplier-out evaluation shows the spread of performance across companies, and that spread, not the pooled average, is the honest preview of what a new supplier's records will produce.
Small evaluation sets need this most. Anthropic's engineering guide to agent evals says "20-50 simple tasks drawn from real failures is a great start." The advice holds, but if all 40 tasks come from one supplier's failures, the result describes that supplier, and the reported interval should say so.
Coverage claims that supplier clustering quietly breaks
Five common claims fail silently when records cluster by supplier. The table pairs each with what clustering hides, the evidence that would support it and wording the evidence allows; use it as the coverage limitations table in a dataset card or an acceptance memo. X, N, A and B stand for your own measured values.
| Claim | What clustering hides | Evidence needed | Wording the evidence allows |
|---|---|---|---|
| The dataset covers the workflow | Steps the dominant supplier never records | Workflow steps present per supplier, not pooled | Covers steps X and Y at N of 6 suppliers |
| Accuracy is X percent, plus or minus 1 point | An interval computed as if records were independent | A supplier-clustered interval and per-supplier scores | Accuracy ranged from A to B across held-out suppliers |
| Rare cases occur at X percent | One supplier's policy producing most rare cases | Rare-case rate per supplier, with counts | Rare cases came mostly from one supplier; the pooled rate is not a market rate |
| The model generalizes to new companies | Test suppliers that also appear in training | Leave-one-supplier-out results | Tested on N suppliers never seen in training |
| Representative of the industry | Six firms standing in for thousands | A sampling frame of firms and a written selection rule | Drawn from six firms chosen by availability, not a probability sample |
Could a large record count still settle the question?
Only when the records come from many suppliers. A large count from a few companies can hide three problems at once: repeated cases, bias in which companies took part, and failures no participating company records.
Repeated cases are the mildest. Near-duplicates inflate counts and should be removed, but clustering survives deduplication, because distinct cases from one company still share its templates, staff and policies. Selection is harder: companies that volunteer records tend to be the ones with tidy systems and long retention, so a pooled dataset can describe well-run firms and miss the rest.
Public benchmarks show the scale question plainly. SWE-bench contains 2,294 software engineering problems drawn from real GitHub issues and pull requests, but they come from 12 popular Python repositories; for a question about how coding agents handle unfamiliar codebases, the number of repositories is the sample size that sets the limit. The same reading applies to a support corpus of 200,000 tickets from four vendors.
Unobserved failures are the hardest. If no participating supplier logs a failure type, such as a customer who left without complaining, no record count reveals it, and the coverage table above is where that limit gets written down.
Caps, weights and spread: limiting any one source's pull
Three design choices keep one supplier from dominating: a per-supplier cap, weights that restore the target mix, and deliberate spread across companies at sourcing time. Fix them before training, because choosing them afterward invites tuning to the result.
Caps have a documented precedent. Datasheets for Datasets, the documentation framework from Gebru and colleagues, describes a movie-review sample capped at forty reviews per unique author to achieve broader coverage by authorship. The same rule applied to suppliers, for example no more than a quarter of training records from any one company, stops a large supplier's habits from becoming the model's habits.
Spread is the stronger fix, and benchmark builders pay for it. OpenAI says it recruited GDPval's experts for breadth, such as lawyers from different practice areas and firms of different sizes, to make the benchmark representative; its 1,320 tasks cover 44 occupations. For a licensed dataset, spread means sourcing more companies at modest volume rather than more volume from the companies already in hand.
Weights repair the mix but not the information. Upweighting a small supplier makes its few records count more in an average, which restores representation but widens the interval, because the effective sample is still set by how many independent sources exist.
Putting supplier structure into a data request
Write the supplier requirements into the request itself: the minimum number of contributing companies, the maximum share from any one, the codes you need carried through preparation and the held-out-supplier test you will run at acceptance. A seller can then plan sourcing around the structure instead of discovering it at review.
SourceX sources datasets for AI buyers on request from U.S. businesses rather than holding stock, and buyers describe scope, volume, language, format, timing and licensing needs through the request form. Adding a supplier count, a per-supplier cap and an acceptance test gives that sourcing a target to plan against; it does not imply that any particular mix of suppliers is available.
A supplier-aware request can include these clauses:
- Minimum contributing companies for the claim you need, with the reason, such as a target effective sample size.
- Maximum share of records from any one company, separately for training and evaluation sets.
- A pseudonymous supplier code, plus site and case codes, on every record.
- The pilot design you will use to estimate the ICC and the effective sample size.
- The held-out-supplier test and the range of per-supplier scores you will accept.
- The coverage limitations table you expect delivered with the dataset.
Frequently asked questions
What is a clustered sample in an AI training or evaluation dataset?
It is a dataset whose records arrive in groups, such as all the records from one supplier company, one site or one agent, where records within a group are more alike than records across groups. Treating them as independent overstates how much evidence the dataset holds and understates the uncertainty in any result.
How do I calculate the effective sample size for records from a few companies?
Divide the record count by the design effect, approximately 1 + (m - 1) × ICC for equal-sized suppliers, where m is records per supplier and ICC is the intra-class correlation for your metric. With 8,000 records per supplier and an illustrative ICC of 0.02, the design effect is about 161, so 48,000 records act like about 300 independent cases.
Is supplier clustering the same as duplicate data?
No. Duplicates are copies of the same case and should be removed. Clustering survives deduplication, because distinct cases from one company still share its templates, staff, customers and policies, so they still carry less independent information than cases from different companies.
How many suppliers does a business dataset need?
It depends on the claim and the ICC. When suppliers each contribute many records, the effective sample size approaches the number of suppliers divided by the ICC, so a target of about 1,000 effective cases at an illustrative ICC of 0.02 needs at least 20 suppliers, and more in practice because each supplier's records are finite.
Should evaluation splits be made by supplier or by record?
Split by supplier whenever the claim is that a model works for companies it has not seen, using group splits such as leave-one-supplier-out. Record-level splits within suppliers answer a narrower question, how well the model does on more records from the same companies, and should be reported as that.
Sources
- Anthropic recommends clustering standard errors on the unit of randomization when eval questions are not independent, such as several questions about the same passage. Source
- Adding Error Bars to Evals (arXiv 2411.00640) shows that, analyzing the same data, clustered standard errors can be over three times larger than naive standard errors, and recommends clustered standard errors when questions come in related groups. Source
- Anthropic's engineering post on agent evals defines tasks, trials and graders and says that 20-50 simple tasks drawn from real failures is a great start. Source
- The UN handbook Designing Household Survey Samples defines the design effect as the ratio of an estimate's true sampling variance to that of a simple random sample of the same size, says the clustering component depends on the subsample size per cluster and the intra-class correlation, and warns against adopting another survey's design effect. Source
- WILDS curates 10 datasets of real-world distribution shifts; in Camelyon17-WILDS the domain is which of five hospitals supplied a patch, and training and test patches come from separate hospitals. Source
- The WILDS documentation describes metadata containing domain identity, such as which hospital a patient's data came from, and Grouper objects that extract group annotations from that metadata. Source
- scikit-learn's cross-validation guide says grouped data breaks the i.i.d. assumption, that GroupKFold ensures the same group is not in both training and test sets, and that LeaveOneGroupOut holds out each group in turn. Source
- SWE-bench comprises 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Source
- Datasheets for Datasets describes a movie-review sample capped at forty reviews per unique author to achieve broader coverage by authorship, and asks how dataset samples were drawn and whether they are representative. Source
- OpenAI's GDPval has 1,320 tasks across 44 occupations, and OpenAI says it recruited a breadth of experts, such as lawyers from different practice areas and firms of different sizes, to maximize representativeness. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.