Definitions and comparisons
Dataset coverage ratio: a 7-line card for verifying vendor claims
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Verify a vendor's coverage claim by turning its count into a dataset coverage ratio you define: a unit, a dated universe, a minimum count per cell, a counting basis, a weighting and a verification sample. Headline counts shrink fast under thresholds: Common Voice's v27.0 release lists 295 languages, of which 187 have at least 10 validated hours and 39 at least 100. Compare vendors only on your own denominator.
Checklist
Dataset coverage ratio
Key takeaways
- A coverage count is only a numerator; the claim becomes checkable when it names the universe it divides by and that universe's version.
- A cell counts as covered only above a stated minimum of verified examples, and evaluation needs a higher minimum than training.
- Report coverage two ways, as a share of cells and as a share of production volume, because the two can rank vendors differently.
- Run a blind relabel sample before counting cells, since dataset metadata on public hosting sites has proved unreliable.
- Restating two vendors on one denominator can reverse the ranking their headline counts suggest.
What does a dataset coverage ratio have to state before you can check it?
It has to state a ratio, not a count. "Covers 320 intents" or "300 languages" is a numerator; the claim becomes checkable only when the vendor also says what one covered unit is, which full universe it divides by, how many verified examples make a unit count, which records were counted and whether the ratio is weighted by how often each unit occurs.
Without those parts, two vendors can both be accurate and still be impossible to compare. One counts every category with a single example, the other only categories with a hundred reviewed ones; one divides by its own taxonomy, the other by an industry code set that changes every year. The buyer's job in verifying a coverage claim is to restate both on one denominator the buyer defines.
This guide shows how fast a public headline count shrinks under thresholds, where the denominator should come from, how to set a minimum per cell, how weighting changes the answer, a seven-line card that makes any coverage ratio reviewable, and a worked comparison of two illustrative vendors. Whether a dataset is representative of a deployment population is a related question covered in a separate SourceX guide; this one is about the arithmetic of the claim itself.
One public dataset, six honest language counts
Common Voice is a useful test case because the project publishes everything needed to compute a coverage ratio, and its own numbers show how much the answer depends on the threshold. Its Scripted Speech release v27.0, dated September 11, 2026, lists 295 languages, 42,593 total hours and 29,295 validated hours, according to the project's dataset repository.
Computed from that release's statistics file, the language count depends entirely on the minimum chosen: 289 languages have any validated audio, 264 have at least one validated hour, 187 have at least 10, 39 have at least 100 and 8 have at least 1,000. The ten largest languages hold 64 percent of validated hours. Every one of those numbers is true; they answer different questions, and only the buyer's use decides which one matters.
Two more details from the repository show why the counting basis matters. Validated hours are about 69 percent of total hours in the release, and the repository notes that demographic splits such as gender, age and accent are computed on the entire dataset, not only the validated set, so a speaker share quoted from those splits describes every recorded clip, including clips that never passed validation. None of this is a criticism of Common Voice, whose openness is what makes the check possible; it is the check a buyer should be able to run on any vendor.
Whose denominator: the vendor's taxonomy or your task population?
Yours. Coverage should be divided by the units your model will meet, listed from your own production records and dated, because a vendor's taxonomy is built around the data the vendor has, and whoever defines the universe can make any catalog look complete.
Build the universe from a fixed window of production data: the intents in twelve months of tickets, the procedure codes billed last year, the document types your agents actually open. Keep an explicit row for other or unknown units, because the cases nobody has named yet are often the ones a model fails on, and they belong in the denominator.
Version the universe as carefully as the data. Standard code sets move: CMS tables for ICD-10-PCS procedure codes show 79,193 codes for fiscal 2026 and 79,258 for fiscal 2027, after 103 additions and 38 deletions. A claim of near-complete coverage of a code set that does not say which year's set it means cannot be checked, and every new code starts at zero examples.
Mapping is where most disagreements hide. A vendor category that merges two of your units should count toward neither until its records are split, because the whole merged count could sit in one of them; a vendor category that splits one of your units can simply be summed. Write the mapping down as a table both sides sign, and count unmapped units in both directions rather than letting either side quietly drop them.
How many verified examples make a cell covered?
Set the minimum by what the data will be used for, write it into the claim, and report coverage in frequency bands rather than as covered or not. Under a count of one, a cell with three examples and a cell with three thousand look the same.
Public datasets already report this way. LVIS, a large-vocabulary instance segmentation dataset, groups its categories by how many training images contain them: rare categories appear in 1 to 10 images, common ones in 11 to 100 and frequent ones in more than 100 (Gupta and colleagues, 2019). Asking a vendor for the same three bands on your universe shows at once whether its coverage is deep or merely wide.
Evaluation data needs more per cell than training data, because each cell has to support its own estimate. With 30 test items in a cell and 27 answered correctly, the 95 percent Wilson interval on accuracy runs from about 74 to 97 percent, too wide to separate two models; with 200 items and 180 correct it narrows to about 85 to 93 percent. Choose the minimum that makes the per-cell interval narrow enough for the decision the cell has to support.
Share of cells or share of volume: two ratios that rank vendors differently
Report both, because they answer different questions. The share of cells covered says how much of the task's variety the data reaches; the share of production volume that falls in covered cells says how much of the model's daily work it reaches. A vendor strong on common cases can win the second ratio and lose the first.
Neither ratio says where the gaps are. Model Cards for Model Reporting (Mitchell and colleagues, 2019) argued for reporting evaluation results broken out by groups and by intersections of groups, because one aggregate can hide poor performance on a subgroup. The same logic applies upstream: a coverage claim should arrive with the per-cell table it summarizes, including the intersections your use depends on, such as segment by channel.
Volume weights also drift. If the production mix shifts, volume-weighted coverage moves even though the vendor's data has not changed, so date the weights and recompute them when the mix moves.
The coverage ratio card: seven lines any claim should fill
The card turns a headline count into a ratio a reviewer can recompute. The vendor fills the entry column for its own claim; the buyer runs the check in the last column before the number goes into any comparison. A line left blank means the claim does not cover it.
Methods note: the card is an editorial framework built for this article on 2026-10-26. Its entries are illustrative and describe no real vendor; the checks draw on the Common Voice repository, the LVIS frequency bands and Model Cards, cited above. The seven lines:
| Line | Question it answers | Illustrative entry | Check the buyer runs |
|---|---|---|---|
| Unit | What counts as one covered thing? | One customer-support intent | Map the vendor's units to yours one to one |
| Universe and version | What does the ratio divide by? | 400 intents from 12 months of tickets, listed 2026-09-30 | Confirm the list is yours, dated and has an unknown row |
| Minimum per cell | How many examples make a cell count? | 50 verified examples for training, 200 for evaluation | Ask for counts in frequency bands, not one cutoff |
| Counting basis | Which records were counted? | Delivered, deduplicated, validated records only | Recount on the delivered files, not the catalog |
| Weighting | Share of cells or share of volume? | Both: share of intents and share of ticket volume | Recompute with your own dated volume weights |
| Verification | Are the labels behind the count right? | Blind relabel of 20 records per covered cell | Apply each cell's agreement rate before counting it |
| Unmapped units | What fell outside the mapping? | 30 vendor categories with no matching intent | List both directions and decide each one by hand |
Two vendors restated on one denominator
Restated on the buyer's universe, the vendor with the bigger headline can come second. In the illustrative comparison below, Vendor B claims 380 intents to Vendor A's 320 and covers more of the buyer's 400 intents at the 50-example minimum, yet after a blind relabel sample A covers slightly more cells and far more of the buyer's ticket volume.
Every number in the table is illustrative and describes no real vendor. The mechanism is general: B's extra cells are thin, and its labels agree with the buyer's rubric less often (81 percent, against 94 percent for A), so many of its cells drop below the minimum once mislabeled records are taken out. A's cells are fewer but deep, and they sit where the tickets are.
The right choice depends on the use. For a router trained on common traffic, A's volume coverage decides it. For an evaluation set meant to probe the long tail, neither vendor alone reaches enough cells, and B's thin cells may be worth paying to relabel. The comparison:
| Measure | Vendor A, illustrative | Vendor B, illustrative | What it tells you |
|---|---|---|---|
| Headline claim | 320 intents | 380 intents | Two numerators on two vendor taxonomies |
| Buyer intents with any examples, of 400 | 290 (72.5%) | 350 (87.5%) | Breadth on the buyer's universe |
| Buyer intents with 50 or more examples | 180 (45.0%) | 230 (57.5%) | Depth before checking labels |
| Same, after the relabel sample | 171 (42.8%) | 160 (40.0%) | Depth on records that match the rubric |
| Ticket volume in cells still covered | 88% | 66% | Reach into the model's daily work |
When the vendor's labels and metadata are wrong
Assume some are, and measure how many before counting anything. A cell is covered only by records that actually belong to it, so the verification sample comes before the coverage ratio, not after it.
Dataset metadata has a weak public track record. The Data Provenance Initiative audited more than 1,800 text datasets and found license omission above 70 percent and error rates above 50 percent on widely used dataset hosting sites (Longpre and colleagues). If a field as basic as the license is missing or wrong that often, category tags, language labels and segment codes deserve the same doubt.
The check is cheap relative to the purchase. Draw a random sample from each covered cell, have someone who cannot see the vendor's label assign one from your rubric, compute the agreement rate per cell, apply it to the cell's count as in the comparison above, and rerun the ratio. Catalog size, security attestations and selected case studies cannot stand in for this sample, because none of them says whether the records in your cells are what their labels claim.
Turning the restated ratio into a purchase decision
Decide from the restated ratios and the per-cell table, and record which cells the purchase is meant to fill. A reviewer should be able to see the denominator, the minimum and the verification behind every number in the decision.
At a glance
Turning the restated ratio into a purchase decision
When records are requested through SourceX, the request is where the denominator gets fixed. SourceX keeps no catalog to count against; it sources records from U.S. businesses for each request, so a request that carries the universe list, its version date, the minimum per cell and the verification method lets every candidate supplier report coverage on the same terms.
The decision memo records, at minimum:
- The universe, its source window and version date, and the unknown row.
- The minimum per cell for each use, with every vendor's counts in frequency bands.
- Both coverage ratios after the verification sample, with each vendor's agreement rate.
- The cells the purchase is meant to fill and the cells it leaves empty.
- The decision: buy, buy for named cells only, test further or stop, and the evidence that would change it.
Frequently asked questions
What is a dataset coverage claim?
It is a vendor's statement about how much of a task's variety its data reaches, such as the number of languages, intents or codes it includes. To be checkable it needs a denominator: the full list of units it divides by, a minimum count per unit and a statement of which records were counted.
What denominator should a buyer use?
The units the model will meet in production, listed from the buyer's own records over a fixed window, dated, and with a row for unknown units. A vendor's taxonomy reflects the data it holds, so dividing by it tends to overstate coverage.
How many examples does a covered category need?
It depends on the use. Training can benefit from modest counts per category, while evaluation needs enough items per category for a usable interval: with 30 items at 90 percent accuracy, the 95 percent interval is about 22 points wide. Ask vendors for counts in frequency bands rather than against a single cutoff.
Does a larger dataset mean better coverage?
Not necessarily, because volume tends to pile up in a few common categories. In Common Voice's v27.0 release, the ten largest of 295 languages hold 64 percent of validated hours. Coverage depends on how records spread across your cells, not on the total.
Can coverage be verified without seeing every record?
Yes. Per-cell counts, frequency bands and a blind relabel sample of about 20 records per cell let a buyer estimate verified coverage without reviewing the whole dataset. Apply each cell's agreement rate to its count before deciding whether the cell is covered.
Sources
- The Common Voice dataset repository lists Scripted Speech v27.0 (2026-09-11) with 295 languages, 42,593 total hours and 29,295 validated hours, and notes that demographic splits are applied to the entire dataset, not just the validated set. Source
- Common Voice's v27.0 statistics file gives validated hours per language, from which 289 languages have any validated audio, 264 at least 1 hour, 187 at least 10, 39 at least 100 and 8 at least 1,000, with the ten largest holding 64 percent of validated hours (this article's computation). Source
- CMS's ICD-10 MS-DRG Version 44 errata tabulates ICD-10-PCS codes at 79,193 for fiscal 2026 and 79,258 for fiscal 2027, after 103 additions and 38 deletions. Source
- LVIS groups categories by the number of training images containing them: rare (1 to 10), common (11 to 100) and frequent (more than 100). Source
- Model Cards for Model Reporting (2019) proposes reporting evaluation results disaggregated across groups and intersections of groups. Source
- The Data Provenance Initiative audited more than 1,800 text datasets and found license omission above 70 percent and error rates above 50 percent on widely used dataset hosting sites. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.