Dataset language coverage validation: a reproducible English-share audit

How do you prove the English share of a dataset with a sample someone else can repeat?

Freeze the exact list of records in scope, publish a fingerprint of that list, select the sample with a rule that anyone can recompute from the record IDs and a stated seed, label every sampled record under a written counting rule, and report a weighted estimate with its interval per stratum and overall. If a second person can rebuild the same sample IDs and land within the stated interval, the figure is substantiated.

This page is about that last step: turning a language-share figure into evidence another party can check. Choosing a unit, picking a language identification model and sizing a first sample are earlier decisions, and the guide to language identification QA covers how reliable those model labels are. Here the question is narrower and comes later in a deal. A supplier has a number, a buyer or reviewer has to rely on it, and both sides need a method that gives the same answer when someone else runs it.

The output is an audit packet: a frame manifest, a selection rule, a stratum table, a labeling log, an estimate with uncertainty, and a replication note. The template further down lists every field.

Why a share computed once on a laptop rarely survives review

Most language-share figures fail review for reasons that have nothing to do with the language model. The record list changed between the measurement and the offer, the sample was drawn with a seed nobody wrote down, or the reported percentage pooled strata of very different sizes as if they were equal.

Automatic labels are also less reliable than test-set accuracy suggests. Caswell and colleagues found that language ID models scoring well on held-out test sets still produced web-crawl corpora with very low human-judged accuracy for many lower-resource languages, and they traced the gap to domain mismatch, class imbalance and similar languages. Kreutzer and colleagues manually audited 230 language subsets of five large multilingual corpora and found mislabeled data and nonstandard language codes. Business exports share the same weak points: short messages, product codes, pasted logs and templates.

The lesson for a supplier is practical. A share that rests only on a model run cannot be defended. A share that rests on a frozen frame, a recomputable sample and human-confirmed labels can.

Freeze the frame before you draw a single record

The sampling frame is the complete list of records the figure describes, and it has to be fixed before sampling. Export one row per record with a stable record ID, the stratum value and the date range filter that produced it, sort it by ID, and compute a SHA-256 digest of the sorted file. Record the row count and the digest in the packet.

That digest is the anchor for everything else. If the frame changes, because a filter is edited, a brand is removed or de-identification drops records, the digest changes and the audit has to be rerun on the new frame. Python's hashlib documentation lists SHA-256 among the constructors always present in the module, so any reviewer can recompute it without special tools.

Decide exclusions before freezing, not after. Records with no linguistic content, such as attachment-only tickets or system events, should be either excluded from the frame by a written rule or kept and reported in their own bucket. Changing that choice after seeing results is the most common way an honest audit turns into a disputed one.

Which strata and selection rule let a second team draw the identical sample?

Use a keyed hash of each record ID as its random number, then take the lowest values within each stratum. Concretely, compute SHA-256 of the audit seed joined to the record ID, read the digest as a number, sort each stratum by it, and select the first n records. Anyone with the frame and the seed gets the same IDs in the same order, on any machine.

A seeded pseudo-random generator is weaker evidence than it looks. Python's own documentation says reusing a seed reproduces the same sequence from run to run, but it also says most of the random module's algorithms and seeding functions may change across Python versions. A sample drawn with a seed alone is reproducible only if the reviewer also matches the language version, library and the order in which records were read.

The hash rule has a second benefit. Because each record's rank does not depend on the others, you can enlarge a stratum later by taking the next records in hash order, and the original sample stays a subset of the larger one. That makes top-up sampling clean when an interval comes back too wide.

Stratify on the variables that drive language, not on whatever columns are convenient. In operational records the mix usually changes by channel, region or brand, customer segment, and year, and sometimes by source system after a migration. Three to six strata is enough for most audits; more strata mean more small cells with wide intervals.

Allocate the sample so every stratum you intend to report on its own has enough records to give a usable interval, then weight the results back to the frame. Proportional allocation is simplest to explain. Oversampling a small but language-diverse stratum, such as web forms from a non-English market, is often worth it because that stratum usually carries most of the uncertainty.

Write the stratum definitions as rules a script can apply, such as channel equals chat, and include the frame count for each. A reviewer should be able to recount every stratum from the frozen file.

Labeling sampled records so the human step can be checked too

Every sampled record gets a final label from a person, with the model output kept beside it. Run a pinned language ID model first so reviewers start from a suggestion, but treat the human label as the measurement. The fastText documentation, for example, distributes its lid.176 models for 176 languages and states they expect UTF-8 input, so record the model file, its version and the encoding step in the log.

Use one written counting rule for mixed-language records and apply it the same way for the whole sample, for example a record counts as English when the majority of its customer-written and agent-written characters are English. Store language labels as standard tags, such as en or es under RFC 5646, so that tags like und for undetermined are unambiguous across tools.

Have a second reviewer label a blind subset, at least every record where the model and the first reviewer disagree and a random slice of the rest. Report the agreement rate and how disagreements were settled. If agreement is poor in one stratum, the counting rule is probably unclear there, and the fix is to tighten the rule and relabel, not to average the disagreement away.

The stratified language audit template

Keep one packet per offered scope. Each field below is something a reviewer can check without asking the person who ran the audit. Example values are illustrative.

How should uncertainty be reported for each stratum and overall?

Report each stratum's share with its own interval and the weighted overall share with an interval that reflects the stratified design. The overall estimate is the sum of each stratum's share times that stratum's fraction of the frame, and its variance is the sum of each stratum's squared weight times the variance of its share, with a finite-population correction when a stratum's sample is a noticeable part of it.

For single-stratum intervals, prefer a method that behaves well near 0% and 100%, because English shares in business data often sit near the top. The statsmodels proportion_confint function, for example, offers Wilson, Clopper-Pearson and Jeffreys intervals alongside the normal approximation, and its documentation notes that Clopper-Pearson has coverage at least the nominal level but is generally conservative. Name the method you used.

Never report only the unweighted pooled share. If one stratum was oversampled, pooling counts gives that stratum too much influence. In the worked example below the pooled figure and the weighted figure differ by a full percentage point.

Worked example (illustrative): a support export that clears 90% only after narrowing

Illustrative example, invented to show the method; it does not describe a real company or dataset. A supplier freezes a frame of 240,000 closed support records split into email (120,000), chat (80,000) and web form (40,000), and samples 300, 250 and 200 records by hash rank. After model suggestions and two-reviewer labeling under a majority-of-characters rule, 285 email, 221 chat and 170 web form records are English.

Per stratum that is 95.0% for email (Wilson 95% interval 91.9% to 96.9%), 88.4% for chat (83.8% to 91.8%) and 85.0% for web forms (79.4% to 89.3%). Weighted by frame share, the overall estimate is 91.1% with a 95% interval of 89.1% to 93.1%. The unweighted pooled share would read 90.1%, which understates email's weight.

Suppose the agreed threshold is that the share must be at least 90%. The point estimate clears it, but the interval does not, so the honest status is consistent with the threshold but not demonstrated. The supplier has two options. It can top up the web form and chat strata by taking the next records in hash order, which keeps the original sample intact. Or it can narrow the offered scope to email and chat, which gives 92.4% with an interval of 90.2% to 94.5% on a new, smaller frame with its own digest. Either choice is recorded in the packet before any offer relies on it.

A second party then reruns the selection from the frame and seed, matches every sampled ID, relabels a blind subset and lands inside the stated interval. Only at that point does the packet move from measured estimate to independently verified.

Defects that change acceptance even when every file opens

Several problems leave the data perfectly readable but invalidate the audit. Each one has a specific check.

• 1Frame drift: the delivered export has a different count or digest than the audited frame. Rerun on the delivered frame; a share does not transfer between frames.
• 2Boilerplate inflation: signatures, macros or quoted history in English push mixed records over the line. Strip them by a written rule before labeling, and say so in the counting rule.
• 3Post-audit filtering: de-identification or exclusion removes more records from one stratum than another, shifting the weights. Recompute weights on the final frame.
• 4Silent relabeling: the model version or threshold changed between runs. Pin the version in the log and treat a change as a new audit.
• 5Undetermined records hidden in the denominator: records with no linguistic content counted as non-English, or quietly dropped. Report them as their own bucket.
• 6Unit mismatch: the threshold was set in characters or tokens but the audit counted records. State the unit next to the threshold and measure in that unit.

When the audit should stop and the scope should change

Stop and redefine the scope when one stratum's interval sits wholly below the threshold, when reviewer agreement in a stratum is too low to trust the labels, or when the frame cannot be frozen because the source system keeps changing. Topping up the sample will not fix any of those.

Bring in a specialist when the data mixes closely related languages or scripts, when audio or scanned documents need transcription before labeling, or when the counting rule itself is contested between the parties. These are judgment calls about measurement design, and they should be settled in writing before the next run.

For a supplier considering a data license, eligibility rules sometimes depend on this number. SourceX, for instance, works with U.S., Canadian and EU companies that had 50 or more full-time employees at their peak, not counting contractors, and considers companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia only when the offered data is at least 90% English, a share it confirms on the dataset actually offered rather than on the company's country or working language. The first step there is a short company assessment that needs no data upload, so an audit packet like this one is preparation for a later stage, not an entry requirement.