AI uses for records
Dataset customer concentration coverage: when one customer dominates
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Tag every record with its end customer, then report each customer's share of records and complete cases, the top-1 and top-5 shares, a Herfindahl-Hirschman Index over those shares and the effective number of customers. Compare the shares with the population the model will serve, list who is missing, and test the model on customers held out of training before trusting any average.
System export
Dataset customer
Key takeaways
- Concentration is measured against the population the model will serve, so write that population down first.
- Count records, complete cases and mature outcomes separately; each can show a different dominant customer.
- An HHI over customer shares, converted to an effective number of customers, states how large the real sample is.
- Antitrust concentration bands were built for merger review and are not data quality thresholds.
- Customer-grouped splits are the direct test of whether a model learned the task or the dominant customer.
How do you measure concentration around a single large customer in training data?
Tag every record with the end customer it belongs to, then report each customer's share of records and of complete cases, the top-1 and top-5 shares, a Herfindahl-Hirschman Index over those shares and the effective number of customers it implies. Compare the shares with the population the model will serve, and test performance on customers held out of training. One number is never enough; the report is the answer.
Dataset customer concentration is a coverage problem, not only a volume problem. A company's support tickets, work orders or contract files can hold tens of thousands of records and still describe mostly one client's products, templates and habits. A model trained on that archive learns the client as much as the task.
This page sets out how to measure that concentration, how to describe who is missing, and how to decide whether to rebalance, disclose or narrow the intended use. The worked figures are illustrative percentages and counts, not results from any dataset.
What one dominant account teaches a model by accident
A dominant customer contributes its own vocabulary, product names, ticket categories, document templates, escalation paths and service terms. When it supplies a large share of examples, a model can score well by learning those patterns rather than the general task, and the weakness only shows when the model meets a different customer.
The effect is worse when the dominant customer also appears in the test set. Records from one customer are not independent: they share people, products and phrasing. The scikit-learn documentation describes exactly this situation for grouped data, noting that the usual independence assumption breaks when samples come in dependent groups, and that measuring generalization to unseen groups requires validation folds whose groups do not appear in training at all.
Concentration also hides outcome bias. If the large account had a dedicated team, faster response targets or a custom workflow, its tickets resolve differently from everyone else's, and a model learns resolution patterns that most customers never receive.
Start with the population the model will serve
Concentration only means something relative to a target. Before counting, write down who the model is for: which kinds of customers, sizes, channels, regions and time period. A dataset that is half one hospital system may be fine for a model deployed only at that system and badly skewed for a model sold to clinics in general.
Dataset documentation practice already asks this. The Datasheets for Datasets questions ask whether a dataset is a sample from a larger set, what that larger set is, whether the sample is representative of it and how representativeness was checked, and what sampling strategy was used. For high-risk systems in the EU, Article 10 of the AI Act requires training, validation and testing data to be relevant and sufficiently representative in view of the intended purpose, and to take account of the specific geographical, contextual, behavioral or functional setting where the system will be used.
Record the population statement and the sampling frame in the report itself, including which systems, date ranges and record types were in scope and which were excluded and why.
Records, cases, customers: which unit to count
Count more than one unit, because each answers a different question and each can show a different concentration. The same archive can look balanced by customer count and badly concentrated by records.
| Unit | What its shares reveal | Common trap |
|---|---|---|
| Records | Which customer dominates the raw training signal | Chatty customers with many short messages inflate their share |
| Complete cases | Which customer dominates finished, labeled tasks | Open or abandoned cases from small customers drop out silently |
| Cases with mature outcomes | Which customer dominates the labels a model is graded on | Recent cases from new customers have no outcome yet |
| Distinct users or sites | Whether one customer's records come from many people or a few | A single administrator can generate most of a customer's records |
| Segment or channel | Whether a customer type or intake channel dominates | Customers grouped under a parent account hide their real spread |
The case and customer concentration report
The report below is the asset: a fixed set of measures a supplier or an internal data team fills in for every dataset, each with the decision it supports. It is an editorial template, version 1.0 dated 2026-10-18. Illustrative values are invented, and the decision column offers starting points for discussion, not standards.
At a glance
The case and customer concentration report
| Report item | How it is computed | Illustrative value | Decision it informs |
|---|---|---|---|
| Population and sampling frame | Written statement of intended users, systems, dates, exclusions | Mid-size property clients, 2022 to 2025, tickets and work orders | Whether any later measure can be judged at all |
| Top-1 customer share | Largest customer's records divided by all records, per unit | 38 percent of records, 22 percent of complete cases | Whether to cap, reweight or disclose |
| Top-5 customer share | Five largest customers' combined share, per unit | 61 percent of records | Whether the long tail is large enough to learn from |
| HHI over customer shares | Sum of squared percentage shares | About 1,620 on records | Tracks concentration between deliveries |
| Effective number of customers | 10,000 divided by the HHI | About six customers | Plain-language size of the real sample |
| Coverage by segment and channel | Shares by customer type and intake channel versus the population | Phone intake 9 percent of records, about a third of target volume | Which slices need sourcing or a warning |
| Missingness by customer | Share of expected fields empty, per customer | Resolution codes absent for 40 percent of small accounts | Whether label quality differs by customer |
| Temporal shift | Customer shares by year or quarter | Largest client grows from 15 to 52 percent over the period | Whether recent data or older data represents deployment |
| Grouped-split performance | Metric on held-out customers versus random split | Accuracy on unseen customers clearly below random-split accuracy | Whether reported quality will hold in deployment |
Reading HHI and the effective number of customers
The Herfindahl-Hirschman Index is the sum of squared shares. The U.S. Department of Justice, which uses it to describe market concentration, gives the example of four firms with shares of 30, 30, 20 and 20 percent, for an index of 2,600; the index approaches zero with many equal-sized firms and reaches 10,000 when one firm holds everything.
Applied to a dataset, replace firms with customers and market share with each customer's share of records or cases. Dividing 10,000 by the index gives the effective number of customers: the number of equally sized customers that would produce the same concentration. An archive from 400 customers with an index of 2,000 behaves, for concentration purposes, like an archive from about five.
Do not borrow the antitrust thresholds. The Department's bands for moderately and highly concentrated markets were set for merger review, not for training data, and there is no published standard for how concentrated a dataset may be. Set your own limits from the deployment population and the grouped-split results, and report the index so it can be compared between deliveries.
Which customers and cases are missing from the archive?
The customers that are underrepresented relative to intended users are usually the ones the archive recorded least: small accounts, churned customers, customers served by phone or in person, and customers whose records were deleted under a retention policy. None of these show up in a share table, because a share table only counts what is there.
Find them by reconciling the archive against an independent list. Billing or CRM account lists show every customer the company served in the period; comparing them with the customers present in the dataset gives a coverage rate by segment. Where the gap is large, record whether it comes from deletion, from a channel that was never logged, or from selective recording, such as only escalated cases being written up.
Note outcome maturity at the same time. Cases opened late in the period, or by new customers, may not have reached a final outcome, so their labels are missing or provisional. Report the share of immature cases by customer so a buyer can see whether the newest customers are effectively absent from the labeled set.
Does the model work for customers it has never seen?
Test it directly with customer-grouped splits. Assign every record a customer group, hold out whole customers for validation and testing, and compare the result with a random split. The scikit-learn GroupKFold splitter, for example, ensures the same group is never represented in both the training and test sets.
Then report results by group rather than as one average. The WILDS benchmark, built around real distribution shifts such as across hospitals and camera locations, reports, for several of its datasets, a worst-group or worst-region score alongside the average, so a weak subpopulation cannot hide behind a strong mean. The Model Cards proposal makes the same point for documentation: report quantitative results broken down by relevant groups and conditions, and explain why those groups were chosen.
For a concentrated dataset, the most informative split is the dominant customer against everyone else. Train without it and test on it, and train with it and test on the rest. If quality on the rest falls well below quality on the dominant customer, the model has learned the customer.
Illustrative case: a field service archive with one property manager
Illustrative scenario with invented figures. A field service company offers four years of work orders and customer messages from about 400 commercial accounts. One property management firm accounts for 38 percent of records and 22 percent of complete cases; the top five accounts hold 61 percent of records; the index on records is about 1,620, or roughly six effective customers.
The firm's share grew sharply in the last two years after it consolidated its sites, and its work orders use a custom priority scheme the company built for it. Phone-originated jobs, common among small accounts, are mostly missing because they were logged only as a line on the technician's schedule.
The decision in this illustration has three parts. The buyer caps the firm at a fixed share of training records by sampling, keeps all its records in a separate evaluation slice, and asks the supplier to add a sample of schedule-only jobs with their outcomes reconstructed from invoices. The report ships with the dataset, and the buyer's model card states that small, phone-led accounts remain thin.
The counterexample is the same archive offered to a team building a tool only for large property managers. There, the dominant customer resembles the deployment population, the cap would discard relevant data, and the report's role is disclosure rather than correction.
Asking a supplier for the concentration report
Ask for the report before agreeing scope, not after delivery. A supplier can usually compute shares by account from its own systems without sending records, and the report tells both sides whether the dataset fits the intended use or needs a narrower description. Customer identities should appear only as pseudonymous keys; the report needs shares and segments, not names.
Concentration also bears on rights. Records about a large customer's operations may be governed by that customer's contract, including confidentiality or restrictions on reuse, so a supplier should check those terms for its biggest accounts first. Removing names lowers privacy risk but does not by itself create a right to license what a contract restricts.
SourceX can source operational data with a known customer mix for AI teams, drawing on U.S. businesses for each request rather than a stockpile; the request form covers scope, volume, language, format, timing and licensing needs. As a rough benchmark for the supplier side, buyers tend to look for two or more years of steady history and tens of thousands of records in one system, with every deal assessed on its merits.
Frequently asked questions
What is dataset customer concentration?
It is the degree to which one or a few of a company's customers account for the records, cases or labels in a dataset. High concentration means a model trained on the data may learn that customer's products, language and workflows rather than the general task.
Is customer concentration the same as dataset bias?
It is one kind of coverage problem that can produce bias. A dataset can be balanced on demographics or topics and still be dominated by one client account, so concentration needs its own measure alongside other representativeness checks.
What HHI is too high for a training dataset?
There is no published standard. Antitrust bands were designed for market concentration in merger review, so treat the index as a tracking measure and set limits from your deployment population and from how the model performs on held-out customers.
Should the dominant customer be removed from the dataset?
Usually not. Capping its share in training, keeping its records as a separate evaluation slice and disclosing the concentration preserves useful data while limiting the distortion. Removal makes sense only when that customer's records fall outside the intended use or the license.
How do you find customers missing from an archive?
Compare the customers present in the dataset with an independent account list, such as billing or CRM records for the same period. The difference, broken down by segment and channel, shows who is underrepresented and points to the reason, such as deletion or an unlogged channel.
Sources
- The U.S. Department of Justice describes the HHI as the sum of squared market shares, gives a four-firm example of 2,600, and notes it approaches zero with many equal firms and reaches 10,000 for a single firm. Source
- Datasheets for Datasets asks whether a dataset is a sample of a larger set, whether it is representative, how that was validated and what sampling strategy was used. Source
- Article 10 of the EU AI Act requires high-risk training, validation and testing data to be relevant and sufficiently representative for the intended purpose and to reflect the setting of use. Source
- WILDS collects real-world distribution shifts, such as across hospitals and camera traps, and, for several datasets, reports worst-group or worst-region performance alongside the average. Source
- Model Cards propose reporting evaluation results disaggregated by relevant groups and conditions and explaining the choice of metrics and groups. Source
- scikit-learn explains that grouped samples break the independence assumption and that GroupKFold keeps the same group out of both training and test sets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.