AI uses for records
Communication volume dataset bias: a voice-to-decision ratio test
By SourceX Editorial · Updated
9 sections
Short answer
Communication volume dataset bias happens when the people and threads that produce the most text dominate a conversation dataset, regardless of how many decisions they actually made. Test it by comparing each author group's share of words with its share of decisions in the system of record. Then sample by decision, cap words per author, and list the decisions that leave no text at all.
Call to transcript
Communication volume dataset
Key takeaways
- Sampling by message or token hands influence to whoever writes most; Pew found the most active 10 percent of U.S. adult Twitter users produced 80 percent of all tweets from U.S. adults in 2019, so workplace archives should be measured, not assumed even.
- Verbosity is not decision relevance: approvals, status changes and phone resolutions can leave little or no text.
- The voice-to-decision ratio divides a group's share of words by its share of decisions; values far from 1 flag over- or underrepresentation.
- Length bias compounds downstream, because LLM judges and reward models have been shown to favor longer responses.
- Draw the sample from decisions in the system of record, attach the surrounding messages, and publish the coverage limits with the data.
What is communication volume dataset bias?
It is the skew that appears when a dataset's selection or weighting follows how much people write rather than what they decide. If records are sampled per message, per token or by thread length, the most prolific writers supply most of the examples, and a model trained or evaluated on them learns their habits, their vocabulary and the decisions they happen to narrate.
Communication volume can be very uneven. Pew Research Center linked survey answers to the accounts of 2,791 U.S. adult Twitter users in 2019 and found that the most active 10 percent produced 80 percent of all tweets from U.S. adults, while the median user tweeted about twice a month. A public social network is not a workplace, and nobody should assume the same split inside a help desk or an approval queue; the point is that the split has to be measured before a dataset claims to represent the work.
A quick screen says whether to worry before the full worksheet. Sort authors by word count, compute the share of all words written by the top 10 percent of authors, then the share of authors who wrote almost nothing. High concentration does not prove bias, because those authors may also make most of the decisions, but it means the choice of sampling unit will change the dataset. Record the screen with the date range and channels it covers, since adding a channel can change the result.
For a buyer, the decision is practical: which records to request, at what unit and with which corrections, so that the dataset teaches the decisions the model will support rather than the voices that filled the archive.
How do prolific writers end up setting the norm?
Skew gets in through four routes, each of which can operate without anyone intending it. Check them separately, because each needs a different fix:
- Message-level sampling units. Drawing a fixed number of messages or chat turns gives each message an equal chance, so, illustratively, an author with 4,000 messages gets forty times the draws of an author with 100.
- Length filters and token budgets. Quality filters that drop short messages, and token budgets that fill up with long threads, remove terse approvals and confirmations first.
- Thread length that tracks trouble. Long threads can mark disputes, exceptions or confusion, so selecting long conversations for richness can over-teach the unusual case.
- Channel coverage. Exports from chat, email and ticket comments miss decisions made by phone, in meetings, in the field or with a button in another system.
Verbosity or decision relevance: what should the sampling unit be?
For any task that supports or makes a decision, the sampling unit should be the decision, with the messages around it attached as context. A decision is something the system of record can see: an approval in NetSuite, a status change in Jira, a resolution code in Zendesk, a dispatch in ServiceTitan. Messages explain those events, but the event is what the model must get right.
Linking needs a key. Ticket comments carry the ticket ID, approval emails often quote a purchase order or invoice number, and work orders carry a job number; chat is harder and may need a time window plus the participants involved. Report the share of decisions that found at least one linked message and the share of messages that linked to no decision, because both numbers bound what the dataset can claim.
Agent evaluation practice draws the same line. Anthropic's guidance on evaluating agents defines the outcome as the final state of the environment at the end of a trial, not what the agent says, and illustrates it with a flight-booking agent whose announcement that a flight is booked matters less than whether a reservation exists in the database. A dataset sampled by words is sampled on the narration; a dataset sampled by decisions is sampled on the state change.
Benchmarks for economically valuable work start from deliverables too. OpenAI's GDPval covers 44 occupations with tasks vetted by professionals averaging more than 14 years of experience, and lists a customer support conversation among its example work products. When the conversation is itself the deliverable, as in reply drafting, message volume can be a sensible unit; when the conversation surrounds a decision, it is not.
Related skews have their own tests: archives tilted toward the strongest employees, toward frequent complainers or toward escalated cases. Volume bias differs because it can appear in a perfectly balanced set of cases, if the text inside each case is drawn by word count.
The voice-share versus decision-share worksheet
Compare each author group's share of words with its share of decisions, then divide the first by the second. A voice-to-decision ratio near 1 means a group's text and its decisions appear in proportion; a ratio of 3 means its words fill three times the space its decisions earn, and a ratio of 0.1 means its decisions are close to silent in the text.
Illustrative: a mid-size services company exports two years of email, chat and ticket comments together with its approval and ticket logs. Account managers write 41 percent of the words but own 12 percent of logged decisions, a ratio of 3.4. Finance approvers write 3 percent of the words and own 22 percent of decisions, a ratio of 0.1. A sample drawn by words would teach an approval model mostly from account managers' requests and almost never from the approvers' answers.
Run the worksheet in this order, with pseudonymous author IDs and role bands instead of names; the illustrative table follows the steps:
- Group authors by role band using pseudonymous IDs, and give bots and automated notifications a group of their own.
- Compute each group's share of messages and share of words on the full export, before any length filter runs.
- Count decisions per group from the system of record, crediting the person who approved, closed, dispatched or resolved.
- Divide word share by decision share for each group, and record the share of decisions with no linked message at all.
- Draw the sample by decision, stratified by decision type, and cap the words any one author can contribute.
- Recompute the ratios on the drawn sample and publish both versions with the dataset.
| Author group (illustrative) | Share of messages | Share of words | Share of decisions | Voice-to-decision ratio |
|---|---|---|---|---|
| Account managers | 34% | 41% | 12% | 3.4 |
| Support agents | 38% | 35% | 46% | 0.8 |
| Escalation specialists | 14% | 16% | 8% | 2.0 |
| Finance approvers | 6% | 3% | 22% | 0.1 |
| Field technicians | 8% | 5% | 12% | 0.4 |
| All groups | 100% | 100% | 100% | 1.0 |
Which decisions leave no text to sample?
Decisions made by click, by phone, in person or by a status change leave little or no text, so a conversation-only dataset understates them however carefully it is weighted. Reweighting cannot recover what was never written; the fix is to pull the decision from another system or to narrow what the dataset claims.
List each silent decision type in the dataset's coverage limitations, with the source that could restore it:
| Silent decision | Where the evidence lives instead | What a text-only sample gets wrong | Source to request |
|---|---|---|---|
| One-click approvals | Approval logs in ERP or procurement systems | Treats the request as the decision and misses who said no | Approval history with approver role and timestamp |
| Phone resolutions | Call disposition codes and call notes | Undercounts quick fixes and overstates written escalations | Disposition codes linked to the ticket |
| Field fixes | Work orders and completion notes | Learns the dispatcher's wording, not the technician's call | Work order outcome and parts used |
| Status-only closures | Ticket status history | Reads silence as inaction | Status audit trail for each ticket |
| Meeting decisions | Minutes and the changes made afterward | Credits whoever wrote the follow-up email | Decision log or the change it triggered |
| Short messages from mobile or second-language writers | The same channels, removed by length rules | Drops whole roles through quality filters | Unfiltered export plus the filter rules applied |
How does length bias compound after the data is collected?
It compounds when the grader or reward model also favors longer text, because a dataset already tilted toward verbose writers is then scored by a judge with the same tilt. The published evidence on that second step is consistent enough to plan for.
Zheng and colleagues reported in 2023 that, when shown a longer and repetitive answer, GPT-3.5 and Claude-v1 judges were biased toward it, while GPT-4 resisted that particular test. The AlpacaEval maintainers caution that automatic evaluators may be biased toward models that generate longer outputs, and say that their length-controlled win rates raised correlation with Chatbot Arena from 0.93 to 0.98. Singhal and colleagues found that reward improvements in RLHF were largely driven by longer responses rather than other features.
Two controls follow. Report the length distribution of examples by author group next to the ratios, and evaluate with length-controlled or outcome-based scoring so that a model cannot earn credit by writing more. Judge models change, so rerun the length check whenever the grader does.
When is the volume skew the signal rather than the bias?
When the task is to write like the archive's best communicators, or when the prolific writers are also the people who decide. A reply-drafting model may need exactly the long, careful messages a few agents produce, and a coordinator role may both write and decide the most, in which case a high voice share is accurate.
Large record counts can still mislead in either case. Templated messages, automated notifications and copied signatures inflate counts without adding decisions, and a few long-running disputes can contribute thousands of messages to a single outcome. Count unique decisions and unique authors before accepting any claim that volume equals coverage.
The decision rule is short: accept the skew when the deliverable is text and the ratio reflects who writes that text well; correct it when the deliverable is a decision; and narrow the dataset's claim when the decision left no text to sample.
What belongs in the dataset card and the data request?
The ratios, the sampling unit and the silent-decision list belong in the documentation that travels with the data. Datasheets for Datasets asks creators to document a dataset's composition and collection process, and Croissant's responsible AI extension provides fields for a description of missing data and a description of biases. Volume skew fits both: the ratio table under composition and biases, the silent decisions under missing data.
A buyer's request can ask for the same evidence up front. SourceX finds datasets for AI buyers among U.S. businesses after a request comes in instead of selling from stock, and its request-data form asks each buyer for the basics: what is in scope, how much volume, which language and format, by when, and under what licensing. Adding the sampling unit, the decision systems to link and an acceptance test lets a supplier say early whether the decision logs exist at all.
Methods note: what this worksheet does not measure
The worksheet, the voice-to-decision ratio and the coverage limitations table are an editorial method built for this article on 2026-10-27. The skew example comes from Pew Research Center's 2019 study; the length-bias evidence from Zheng et al. (2023), the AlpacaEval repository and Singhal et al. (2023); the decision-as-outcome framing from Anthropic's agent evaluation guidance and OpenAI's GDPval; and the documentation fields from Datasheets for Datasets and the Croissant responsible AI specification. Every worksheet value is illustrative.
Not measured: voice-to-decision ratios in any real business archive, how often silent decisions occur, and whether correcting volume skew improves a model on a given task. Those need a dataset, its decision logs and a buyer's acceptance test, and no such results are published here.
Frequently asked questions
How is communication volume bias different from verbosity bias in LLM judges?
Communication volume bias is a property of the data: the people and threads that write most dominate what gets sampled. Verbosity bias is a property of a grader that prefers longer answers. The two compound when a volume-skewed dataset is scored by a length-favoring judge, so check both.
Should prolific authors be removed from a conversation dataset?
Not as a first step, because their messages can carry real context and skill. Cap the words any one author contributes, draw the sample by decision, and keep their records where they made the decision or wrote the deliverable the task needs.
How do I find decisions that left no messages?
Start from the system of record rather than the conversation export: approval logs, ticket status histories, work orders and call disposition codes. Count decisions with no linked message and list each type in the dataset's coverage limitations.
Does a bigger conversation export fix volume bias?
No. More of the same export adds more text from the same prolific writers and repeats the same silent gaps. Volume bias is fixed by changing the sampling unit and adding decision sources, not by adding rows.
Can weighting alone correct the skew?
Weighting can rebalance groups whose decisions appear in the text but are underweighted. It cannot recover decisions that were never written down, and heavy weights shrink the effective sample. When a group's ratio is near zero, add its decision source or narrow the claim.
Sources
- Pew Research Center's 2019 study of 2,791 U.S. adult Twitter users found the most active 10 percent produced 80 percent of tweets from U.S. adults, and the median user tweeted about twice a month. Source
- Anthropic's agent evaluation guidance defines the outcome as the final state of the environment at the end of a trial, illustrated by whether a flight reservation exists in the database rather than what the agent says. Source
- OpenAI's GDPval covers 44 occupations with tasks vetted by professionals averaging more than 14 years of experience, and its example work products include a customer support conversation. Source
- Zheng et al. (2023) found GPT-3.5 and Claude-v1 judges biased toward a longer, repetitive answer, while GPT-4 resisted that test. Source
- The AlpacaEval repository cautions that automatic evaluators may be biased toward longer outputs and reports that length-controlled win rates raised correlation with Chatbot Arena from 0.93 to 0.98. Source
- Singhal et al. found that reward improvements in RLHF were largely driven by increasing response length rather than other features. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source
- The Croissant responsible AI extension defines rai:dataCollectionMissingData for a description of missing data and rai:dataBiases for a description of biases in the dataset. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.