Skip to content

Privacy and preparation

Business dataset English coverage study: a protocol to measure it

By SourceX Editorial · Updated

9 sections

Short answer

This is a study protocol, not a result. To measure English-language coverage in business archives, define English share over a stated denominator, sample archives and then records within them by record type, label language with a machine model checked against a double-coded human sample, correct for the model's error rate and report each share with an interval that reflects clustering within archives. No results are reported here.

Countries and languages

Business dataset English

This page covers business dataset English coverage study: a protocol to measure it.Illustrative

Key takeaways

  • English coverage is not one number: records fully in English, English share of text and records whose task-relevant text is English answer different questions.
  • Sample archives first and records second, stratified by record type, because records inside one archive resemble each other.
  • Language identification models are fast but fail on short, mixed and templated text, so a human-coded subsample must measure and correct their error.
  • Report intervals that account for clustering, per stratum, never a single pooled percentage without its denominator.
  • This page publishes the method only; no measured findings are reported or implied.

How could a study measure English-language coverage in business archives?

By treating it as a survey estimate with a declared population, a sampling plan and a measurement error model. The study picks a population of business archives, samples archives and then records within them, labels each sampled record's language with a model whose errors are measured on a human-coded subsample, and reports English share per stratum with intervals. This page sets out that design; it reports no results.

The reason to measure is that assumptions about English are usually wrong in one direction or the other. A company headquartered in an English-speaking market may hold large non-English archives from subsidiaries or customers, and a company elsewhere may run its operations, tickets and contracts largely in English. Neither the country nor the interface language answers the question; the records do.

Three questions are in scope: what share of records in a defined set of archives is in English, how that share differs by record type, and how far a fast machine estimate departs from a careful human one. Questions about why archives differ, or what they are worth, are out of scope.

What does English coverage mean, and over which denominator?

State the denominator before any count. A study that reports English coverage without saying whether it counted records, characters of natural-language text or task-relevant fields cannot be compared with any other study, and readers will assume the most flattering version.

Label languages with standard codes so results can be joined to other work. RFC 5646 language tags put the primary language subtag first, and two-letter subtags such as en come from ISO 639-1; record a script subtag only where it adds information. Mixed records need a rule, not a judgment call per record.

What does English coverage mean, and over which denominator?
DefinitionNumeratorDenominatorBest for
Record-level EnglishRecords whose natural-language text is entirely or almost entirely English, by a stated ruleAll sampled records with natural-language textDeciding whether records can be used as whole units
Text-share EnglishCharacters of natural-language text identified as EnglishAll characters of natural-language text, excluding code, numbers and identifiersEstimating how much usable English text exists
Task-field EnglishRecords whose task-relevant fields are EnglishRecords with those fields populatedA named downstream use, such as ticket resolution notes
Mixed-record shareRecords with two or more languages above a stated minimumAll sampled records with natural-language textShowing where record-level labels hide mixing

Population, sampling frame and the two-stage sample

Define the population as archives, not companies: a set of business records of declared types held by an organization for a declared period. Companies hold several archives, and an archive is what someone would actually measure or offer. Write down how archives enter the frame, for example volunteers from an outreach list, and accept that a volunteer frame supports only statements about that frame.

Steps

Population, sampling frame and the two-stage sample

Sample in two stages. First draw archives, stratified by the characteristics the research question cares about, such as headquarters region, industry and system type. Then, inside each sampled archive, draw records stratified by record type, because email, support tickets, contracts and system notes often differ in language even within one company. Fix collection dates, and record archives that decline or drop out together with their stratum, so nonresponse can be described.

Plan sample sizes from the interval width the study needs, not from convenience. Decide in advance how narrow the interval on the primary estimate must be for the result to be useful, work out the number of archives and records per stratum that width requires under a stated assumption about how alike records within one archive are, and record that assumption in the registration. If recruitment falls short, report the wider interval that results rather than quietly merging strata to make the numbers look steadier.

  • Unit of analysis: the record, with the archive as the cluster.
  • Archive strata: headquarters region tier, industry group, primary system type, archive size band.
  • Record strata: record type within the archive, such as tickets, email, contracts, notes, documents.
  • Selection inside each stratum: simple random sample of record IDs from a full ID list, never the newest or the first exported.
  • Collection period: fixed start and end dates for drawing samples, declared before drawing.
  • Nonresponse log: archives invited, agreed, measured, withdrawn, each with stratum and reason.

Measuring language: machine labels checked by people

Use a language identification model for every sampled record, then measure how wrong it is. fastText distributes models that recognize 176 languages, trained on Wikipedia, Tatoeba and SETimes text and expecting UTF-8 input. GlotLID, built on the same approach, now supports 2,102 language and script labels. CLD3 can split a text into spans by script and return the top languages with the proportion of bytes each covers, which suits mixed records; it returns unknown when a span is shorter than its minimum byte count.

The weak points are the ones business archives are full of. Short strings are uncertain: fastText's own documentation shows its newer model giving English a probability of about 0.81 for the two-word greeting Hello, world, with other languages behind it. Templates, signatures and system boilerplate in English sit inside records written in another language. And labels at scale can be badly wrong: an audit of 205 language-specific web-mined corpora by Kreutzer and colleagues found at least 15 with no usable text and many mislabeled.

So the protocol adds a human step. Two fluent coders independently label a random subsample of records under the written rules, disagreements are adjudicated, and the model's confusion against the adjudicated labels is used to correct the estimate and widen its interval.

  • Preprocess: strip quoted replies, signatures, disclaimers and system templates by a declared rule, and keep a copy of what was stripped.
  • Segment: split records into paragraphs or script spans; skip spans under the declared minimum length and count them.
  • Machine label: one pinned model version for the whole study, with its threshold declared.
  • Human label: two coders on a random subsample per record stratum, blind to the machine label.
  • Adjudicate: a third coder resolves disagreements; report agreement before adjudication.
  • Correct: adjust stratum estimates with the measured false-positive and false-negative rates for English.

The variables each sampled record carries

Every sampled record should leave the measurement step with the same fields, so that tables can be rebuilt by anyone with the anonymized study file. Keep content out of the study file; it holds measurements, not text.

At a glance

The variables each sampled record carries

The variables each sampled record carries
VariableDefinitionValuesUsed for
archive_id and stratumPseudonymous archive key and its archive stratumKey plus stratum codeClustering and weighting
record_typeRecord stratum inside the archiveTicket, email, contract, note, document, otherPer-type estimates
nl_charsCharacters of natural-language text after preprocessingCountText-share denominator
stripped_charsCharacters removed as templates, signatures or quotesCountShowing how much boilerplate was excluded
machine_langTop language tag and probability per spanLanguage tag plus probabilityMachine estimate
en_char_shareShare of natural-language characters labeled EnglishProportion between 0 and 1Text-share estimate
short_spansSpans below the minimum lengthCountMeasuring what the model could not label
human_langAdjudicated record label, where codedEnglish, other, mixed, undeterminableError measurement
coder_agreementWhether the two coders agreed before adjudicationYes or noReliability reporting
task_field_enWhether the named task fields are EnglishYes, no, not populatedTask-field estimate

How uncertainty is estimated and reported

Report every share with an interval, a sample size and a denominator, per stratum. For a single proportion from a simple random sample, use the Wilson score interval, which the NIST Engineering Statistics Handbook describes and notes is recommended in the statistical literature; unlike the simple normal approximation, it never runs below 0 or above 1, which matters because English shares in single-language archives sit near those edges.

Records within one archive are not independent, so a simple interval across all records is too narrow. Estimate intervals by resampling whole archives within strata, report the number of archives as well as the number of records, and add the uncertainty from the language identification correction. When a stratum has too few archives for a stable interval, say so and do not report a share for it.

  • Primary estimate: English share per record-type stratum, with archive-level resampled intervals.
  • Secondary estimate: the gap between machine-only and corrected shares, with its interval.
  • Agreement: coder agreement before adjudication, per stratum.
  • Sensitivity: results under an alternative mixed-record rule and an alternative boilerplate rule.
  • Baseline: the machine-only estimate is the baseline the corrected estimate is compared against.

What this design would not support

The design estimates language shares in the archives that were measured, for the period measured. It does not license wider claims, and the write-up should say so in the same section as the results.

  • It does not estimate English coverage for all companies in a country or industry unless the archive frame was a probability sample of that population.
  • It does not show why archives differ; region, industry and system type are strata, not causes.
  • It does not say whether English records are suitable, licensable or valuable; language is one property among rights, privacy and quality.
  • It does not measure translation quality, dialect or writing quality.
  • It does not extend to audio, images or scanned documents without a separate transcription or OCR error model.

Pre-registration checklist for an English coverage study

Register the plan before drawing any sample. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan to a public repository before data collection or analysis begins, with later changes noted. For this study, the list below is the minimum the registration should contain.

At a glance

Pre-registration checklist for an English coverage study

Checklist · 10 items

  1. Research questions and the one primary estimate.
  2. Population and frame definition, recruitment route and inclusion rules for archives.
  3. Coverage definitions in use, with numerators and denominators.
  4. Archive strata, record strata and sample sizes planned per stratum, with the reasoning.
  5. Preprocessing rules for templates, signatures, quotes, code and identifiers.
  6. Language identification model, version, threshold and minimum span length.
  7. Human coding rules, subsample size, coder qualification and adjudication process.
  8. Correction method for machine error and interval method for clustered data.
  9. Collection dates, nonresponse handling and the rule for suppressing small strata.
  10. What will be published: tables, the anonymized study file and the code.

0 of 10 ticked. Tick each item as you go.

Running the protocol on one archive before a licensing review

A single company can run the same steps on its own archive as a one-cluster study. The interval will reflect only that archive, which is what matters to the company: the share of English in the records it might actually offer, by record type, with the boilerplate removed and the machine error measured.

That measurement is directly useful in a licensing review. SourceX's eligibility has two tiers: U.S., Canadian and EU companies that peaked at 50 or more full-time employees (contractors excluded), and companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia, which can be considered when the offered data is at least 70% English, confirmed on the actual dataset. The share is taken on the records offered, not the company's country or working language, and the data still goes through rights, privacy, quality and buyer-fit review. Before any measurement is shared, a short company assessment shows whether the business qualifies, and it asks for no data.

Frequently asked questions

Does this page report how much business data is in English?

No. It publishes a study design only. Any figure would need the sampling, measurement and uncertainty steps described here, carried out and disclosed.

Why not just run a language detector over every record?

Detectors are least reliable on short text, mixed records and boilerplate, which business archives contain in quantity. Without a human-coded subsample to measure the detector's error, the result has an unknown bias.

Should English share be counted by records or by characters?

It depends on the question. Record-level shares suit uses that need whole records in English, while character shares estimate how much English text exists. Report the one that matches the use, and always state the denominator.

How should emails with English signatures and non-English bodies be handled?

Strip signatures, disclaimers, quoted replies and templates by a declared rule before measuring, and count what was stripped. Otherwise English boilerplate inflates the English share of records written in another language.

Why do intervals need to account for archives?

Records from the same company resemble each other, so they carry less independent information than the record count suggests. Resampling whole archives gives honest intervals and makes clear how many organizations the estimate rests on.

Sources

  • fastText's language identification models recognize 176 languages, expect UTF-8 input, and its documentation shows an English probability of about 0.81 for a two-word greeting. Source
  • CLD3 can split text into script spans and return the top languages with the proportion of bytes each covers, and returns unknown below a minimum length. Source
  • GlotLID version 3 supports 2,102 labels made of ISO 639-3 codes with script. Source
  • Kreutzer and colleagues audited 205 language-specific web-mined corpora and found at least 15 with no usable text and many mislabeled. Source
  • RFC 5646 language tags begin with a primary language subtag, and two-letter subtags such as en come from ISO 639-1. Source
  • The NIST Engineering Statistics Handbook describes the Wilson interval for a proportion and its recommendation over simpler methods. Source
  • The Center for Open Science describes preregistration as posting a time-stamped, read-only study plan before data collection or analysis, with later changes noted. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies