Systems and records
Dataset sample weighting: detecting complaint bias in support records
By SourceX Editorial · Updated
9 sections
Short answer
To detect complaint bias, count records per customer before anything else. If a small share of accounts produces a large share of tickets or cases, the dataset describes those customers, not your customer base. Decide which unit the buyer's task needs, then cap records per customer, weight to a reference frame or restratify, check the effective sample size the weights leave, and ship the plan with the data.
Table overview
Dataset sample weighting
Key takeaways
- Complaint and support exports record contacts, not customers, so the customers who contact you most can dominate a dataset without anyone choosing them.
- The first decision is the unit of analysis: ticket, customer, issue or conversation turn. Weights only make sense once that unit is fixed.
- A concentration profile (share of records from the top accounts, records per customer, repeat contacts per issue) shows whether weighting is needed at all.
- Weights cost precision: the Kish approximation turns weight variation into a design effect and an effective sample size that should be reported with the data.
- Weighting cannot recover customers or channels that never appear in the records, and for complaint-handling tasks the skew may be the signal, so document it instead of removing it.
How do you detect complaint bias in an operational dataset?
Profile who produced the records before you profile what they say. Group every ticket, case or message by customer account and compute how concentrated the volume is: the share of records from the top 1, 5 and 10 percent of accounts, the median and maximum records per account, and how often one issue spawns several records. If those numbers show a small group carrying a large share of the volume, the dataset overrepresents high-contact customers.
The Datasheets for Datasets framework asks dataset creators a direct version of this question: if the dataset is a sample of a larger set, is it representative, and how was that checked. For operational records the larger set is usually your customer base or your book of transactions over the same period, so the comparison you need is records per customer against customers in the base.
Concentration alone is not a defect. It becomes one when the buyer's task assumes each record stands for a typical customer, issue or interaction. The rest of this plan turns that judgment into explicit choices a buyer can audit.
Why do frequent complainers end up dominating ticket and case exports?
Support, claims and complaint systems create a record each time someone makes contact, so volume follows behavior rather than population. A customer who follows up five times on one problem produces five tickets or one ticket with many messages, depending on how the help desk merges threads. A customer with the same problem who never writes in produces nothing.
Regulators who publish complaint data say this plainly. The Consumer Financial Protection Bureau describes its complaint database as not a statistical sample of consumers' experiences in the marketplace and says the complaints are not necessarily representative of all consumers' experiences with a product or company. The same caution applies to a company's internal ticket history.
Three operational patterns amplify the skew. Large accounts often have more users and more seats, so they file more tickets for ordinary reasons. Escalation workflows create child cases, transfers and reopens that look like new records in an export. And automated sources such as monitoring alerts or form integrations can attach many records to one account. Each needs its own fix, which is why the diagnosis comes before any weight.
Which unit of analysis should the weights describe?
Choose the unit the buyer's model will be judged on, then make every count and weight refer to it. A model that drafts replies is judged per message, a model that routes cases is judged per case, and a model that predicts churn from support history is judged per customer. Weighting tickets when the task is per customer, or the reverse, produces numbers that look tidy and answer the wrong question.
Write the unit into the plan with its key: for example ticket ID for case-level tasks, account ID for customer-level tasks, or a thread key that joins merged and reopened tickets for issue-level tasks. Record the merge and reopen rules from the help desk configuration, because they decide how many records one issue produces.
| Buyer task | Unit of analysis | Key to group on | What overrepresentation distorts |
|---|---|---|---|
| Reply drafting or summarization | Message or conversation turn | Message ID within ticket | Tone and phrasing learned mostly from a few prolific writers |
| Routing and triage | Ticket or case | Ticket ID, with merged tickets collapsed | Category priors tilted toward high-contact accounts |
| Root-cause or issue classification | Issue thread | Thread key joining reopens, follow-ups and child cases | Repeat follow-ups counted as separate issues |
| Customer-level prediction such as churn risk | Customer account | Account ID | Model learns that contact volume itself predicts the outcome |
| Resolution-time estimation | Ticket or case | Ticket ID with status history | Long, escalated cases from a few accounts stretch the distribution |
Which steps make up the representation and sampling weight plan?
Use the plan below as a working document. Each step has something to compute, a decision rule that a reviewer can check and the evidence that should travel with the dataset. Thresholds are placeholders to set with the buyer, not standards. Documentation vocabularies already have a place for the result: the Croissant responsible-AI extension lists customer feedback data as a collection type and provides properties for describing dataset biases and limitations, and it warns that uninformed sampling can bias training data.
| Step | What to compute | Decision rule | Evidence to ship |
|---|---|---|---|
| 1. Unit and scope | Unit of analysis, key, date range, systems and channels included | Unit matches the buyer's evaluation unit; every included channel is named | Scope statement with record and account totals |
| 2. Deduplication | Merged, reopened, child and automated records per issue | Collapse or flag records that describe the same issue before counting | Dedup rule and before and after counts |
| 3. Concentration profile | Share of records from the top 1, 5 and 10 percent of accounts; records per account distribution | If the top accounts carry far more than their share of the base, continue to step 4; otherwise document and stop | Concentration table and histogram data |
| 4. Reference frame | Distribution of the base by segment, plan, region or product from a system of record such as billing or CRM | Frame must cover the same period and population as the records | Frame source, extract date and segment totals |
| 5. Strata | Two to four variables that drive both contact volume and the task, such as plan tier and product | Every stratum has enough records to support a weight; merge thin strata | Stratum definitions and cell counts |
| 6. Weight rule | Per-account cap, inverse-frequency weight, post-stratification or raking to the frame | Choose the simplest rule that removes the skew the task cares about | Weight formula, code and version |
| 7. Trim and precision | Weight range, design effect and effective sample size | Trim extreme weights if the effective sample size falls below what the buyer's tests need | Design effect, effective sample size and trim bounds |
| 8. Sensitivity check | Key task metrics or label shares unweighted, capped and weighted | Large differences mean the skew matters; report all three | Comparison table |
| 9. Disclosure | Limitations that weighting cannot fix, such as silent customers and missing channels | Anything not correctable is stated, not hidden | Bias and limitations text in the dataset card |
How do you set weights without spending too much precision?
Start with the bluntest tool that works. A per-account cap, for example keeping at most a fixed number of randomly chosen tickets per account per quarter, removes extreme concentration without any weights and is easy for a buyer to reproduce. Use weights when you need to keep every record, for example because rare issues come mostly from large accounts.
Inverse-frequency weights give each record a weight proportional to one over its account's record count, so each account contributes the same total weight. For class imbalance, scikit-learn's balanced setting uses the analogous formula: number of samples divided by the number of classes times the count of each class. When you need the weighted records to match known totals on several variables at once, raking, which Pew Research Center's pewmethods package describes as iteratively adjusting the marginal distributions of selected variables to target distributions, is the standard survey approach.
Every weighting scheme trades bias for variance. The pewmethods package computes the Kish approximation of the design effect as one plus the squared ratio of the weights' standard deviation to their mean, and the effective sample size as the record count divided by that design effect. Pew's own comparisons found that a larger design effect came mainly from how far the unweighted sample sat from the population on the weighting variables, which is exactly the situation with heavily concentrated tickets.
Trimming caps the largest weights to recover precision, at the cost of moving the weighted distribution away from the targets; the pewmethods documentation says there is no exact guideline for that balance. Report the untrimmed and trimmed design effect so the buyer can choose.
How do cap, weights and strata compare on an illustrative help desk where a few accounts file most tickets?
All figures in this example are illustrative. A software company exports 24 months of tickets from its help desk for a buyer building a ticket-routing model. The unit is the ticket, with merged tickets collapsed under the surviving ID.
The concentration profile shows that 2 percent of accounts produce 31 percent of tickets, and nearly all of those accounts are on the enterprise tier, which makes up 9 percent of active accounts in the billing system. Enterprise tickets are also far more likely to be categorized as integration issues, so an unweighted routing model would overpredict that category for everyone.
The team tries three versions and compares them in the sensitivity step. A cap of 20 tickets per account per quarter removes 18 percent of tickets and brings the enterprise share of tickets close to its account share. Inverse-frequency weights keep all tickets but produce a design effect of about 2.6, leaving an effective sample size well under half of the record count. Post-stratification on plan tier alone produces a design effect of about 1.3. The team ships the capped set as the default, includes the full set with tier weights as an option and documents the integration-issue share under all three versions.
| Measure (illustrative) | Unweighted | Capped at 20 per quarter | Weighted to plan tier |
|---|---|---|---|
| Share of tickets from enterprise accounts | 31 percent | 12 percent | 9 percent (weighted) |
| Share categorized as integration issue | 27 percent | 16 percent | 14 percent (weighted) |
| Records kept | 100 percent | 82 percent | 100 percent |
| Approximate design effect | 1.0 | 1.0 | 1.3 |
Which weighting defects change a buyer's acceptance even when every file opens?
A dataset can pass every format check and still fail acceptance because its weights do not mean what the documentation says. These are the defects worth testing before delivery.
- Weights computed at the wrong unit: account-level weights attached to messages, so long threads quietly count more.
- Deduplication after weighting: merged or reopened tickets removed after weights were calculated, leaving totals that no longer match the frame.
- A reference frame from a different period or population, such as current active accounts used to weight three years of tickets that include churned customers.
- Strata with a handful of records carrying very large weights, so a few tickets drive the weighted results.
- Weights that depend on a field the buyer will not receive, which makes them impossible to verify or recompute.
- Splits drawn before weighting, so training and test sets carry different concentration and the buyer's evaluation looks better or worse than it is.
- A weight column with no formula, code version or frame extract date, which leaves the buyer unable to tell a design choice from an error.
When is weighting the wrong fix?
If the buyer is building a model to handle escalations, detect complaint risk or assist the people who manage high-contact accounts, the concentration is the phenomenon being learned. Capping or down-weighting those accounts would remove the signal. In that case keep the records as they are, describe the concentration in the dataset card and let the buyer decide how to sample.
Weighting also cannot create customers who never appear. Weights redistribute influence among records you have; they do nothing for customers who called instead of writing, gave up silently or complained on a channel the export does not include. The CFPB makes the parallel point that a low number of complaints about a product or company does not necessarily mean there is little or no consumer harm. Name the missing channels in the limitations text.
Stop and ask for specialist review when the variables that would make good strata are sensitive, such as health, age or protected characteristics, or could re-identify a small account. Narrow the scope when the effective sample size after weighting is too small for the buyer's planned tests; a smaller, cleaner subset with an honest description is better than a large weighted set nobody can rely on.
Why bring a weighting plan to a licensing review?
A concentration profile and a short weighting plan answer one of the first questions a technical reviewer asks about support and complaint records: whose experience does this dataset describe. Preparing it before a licensing conversation also tells you early whether you have enough distinct customers and history to matter, which buyers typically judge in years of consistent records and tens of thousands of records in one system, assessed deal by deal.
Support teams that license ticket and case history through SourceX leave it the sourcing, rights review, contract, data preparation, delivery and payment that turn chosen records into an AI dataset, and the concentration profile can travel with that delivery. Whether the history fits at all is answered first by a short company assessment, with no tickets uploaded.
Frequently asked questions
What is dataset sample weighting?
Dataset sample weighting assigns each record a number that sets how much it counts in training or analysis, so that the weighted records match a reference population or a chosen balance. In operational data it is mostly used to stop high-volume customers, classes or segments from dominating. The weights, their formula and the reference frame should ship with the data.
Should I remove repeat complainers from a support dataset?
Usually not. Capping records per customer or weighting them down keeps their issues represented without letting them dominate. Removing them entirely loses real problems and can bias the data in the other direction, and for escalation or complaint-handling tasks they are the most important records.
How do I know whether weights have cost too much precision?
Compute the design effect and effective sample size. Under the Kish approximation the design effect is one plus the squared ratio of the weights' standard deviation to their mean, and the effective sample size is the record count divided by it. If the effective sample size is below what the buyer's tests need, trim extreme weights or narrow the scope.
Is a per-customer cap better than weights?
A cap is simpler, reproducible and leaves no weight column to misuse, so it is a good default. Weights are better when you must keep every record, for example because rare issue types come mostly from a few large accounts. Many suppliers ship a capped default and a full set with documented weights.
Can weighting fix a dataset that is missing a channel such as phone support?
No. Weights only redistribute influence among records that exist. Customers who used an unexported channel, or never contacted you, are absent, and the dataset card should say so with the channels that are included and excluded.
Sources
- The CFPB says its complaint database is not a statistical sample of consumers' experiences and that few complaints do not necessarily mean little consumer harm. Source
- Pew's pewmethods package computes the Kish design effect as one plus the squared coefficient of variation of the weights and the effective sample size as n divided by the design effect. Source
- The pewmethods weighting vignette defines raking and explains that trimming raises effective sample size while moving weighted distributions away from targets, with no exact guideline for the balance. Source
- scikit-learn's balanced class weights equal the number of samples divided by the number of classes times each class count. Source
- The Croissant RAI vocabulary lists customer feedback data as a collection type, defines dataBiases and dataLimitations properties and warns that uninformed sampling can bias training data. Source
- Datasheets for Datasets asks whether a sampled dataset is representative of the larger set and how that representativeness was validated. Source
- Pew Research Center found a larger design effect arose mainly because the unweighted sample diverged more from population parameters on the weighting variables. Source
See if your company qualifies
A short company assessment. No data uploads are needed.