Privacy and preparation
Dataset language mix validation: test the text, not the country label
By SourceX Editorial · Updated
9 sections
Short answer
Validate a dataset's language mix on the text, not on where the company or its customers are. Sample tasks rather than accounts, label each segment's language and each task's input and output languages, and test every country or locale field against those labels. Keep a proxy only where the sample shows it agrees with the text, and report mixed-language tasks as their own class instead of splitting them into shares.
Countries and languages
Dataset language mix
Key takeaways
- A country, account region or user locale records where someone is or what they chose, not the language a record is written in.
- A record's language has four readings, declared, configured, observed and task, and only the last two describe the text a model will see.
- Mixed-language tasks, such as a Spanish request answered in English, vanish when language is reported as shares, yet multilingual buyers often need them most.
- The proxy agreement test keeps a geography field only for stratifying a sample, and only where labeled text shows it agrees.
- Report the mix by task class with intervals, then compare it with the mix the buyer will deploy into.
Why is a country label not a language profile?
Because geography describes where a company, account or user is, while a language profile describes the text. The two diverge in every direction: U.S. companies answer customers in Spanish, Canadian ones in French and English, and multinational teams write internal notes in a shared working language whatever the customer's country.
Official statistics show how loose the link is. The U.S. Census Bureau reported in June 2025 that more than 1 in 5 people aged 5 and older in the United States, 22 percent in the 2017-2021 American Community Survey, spoke a language other than English at home. A U.S. label on a customer record therefore says little about the language that customer writes in.
Language tags draw the same line formally. RFC 5646, the standard behind BCP 47 language tags, treats a region subtag as a qualifier of a language, as in de-AT for German as used in Austria or es-419 for Latin American Spanish. The language comes first and the region refines it; a country code alone names no language. Dataset language mix validation follows that order: measure the language, then use geography only where it helps.
Four readings of a record's language, and which ones describe the text
Every record carries up to four readings of its language, and only two of them are about the text. Treat the other two as hints for sampling, never as the measured mix.
At a glance
Four readings of a record's language, and which ones describe the text
The configured reading is the one most often mistaken for content. Zendesk's Users API, for example, documents a user's locale as a BCP 47 tag, such as en-US, stored on the user record. It records a profile setting: a customer whose profile says en-US can still write in Spanish, and an agent replying in French may keep an English interface. The four readings are:
- Declared: where the seller says it operates, such as headquarters and office countries. Useful for choosing which systems to sample; silent about any single record.
- Configured: settings attached to accounts and users, such as billing country, user locale, interface language, brand or queue. Useful for stratifying; often stale or left at a default.
- Observed: the language of each text segment, whether a message, note, field or attachment, labeled by a detector and checked by a person where the detector is unsure.
- Task: the languages a task requires, meaning the language of its inputs and the language its output must be in. This is the reading a buyer's model will actually face.
Which mixed-language tasks disappear when the mix is reported as shares?
The tasks that span languages disappear first. A share computed by segment counts a Spanish customer message and an English agent reply as one of each, and the fact that the task required reading Spanish and answering in English is lost.
Research treats code-switching as its own problem for this reason. The LinCE benchmark (2020) combines ten corpora covering four code-switched language pairs, including Spanish-English and Hindi-English, and evaluates tasks such as language identification and named entity recognition on text that mixes languages. Business records carry the same pattern in quieter forms: English product names inside German messages, or a Spanish thread with an English escalation note.
Report task classes alongside language shares. Assign each sampled task to the first class in this order that fits it, so the class counts add up to the sample:
- Language-light: tasks dominated by codes, numbers, templates or signatures, where a detector's label says little.
- Cross-lingual: inputs in one language and a required output in another, such as a German supplier quote summarized as an English purchase note, or a Spanish request answered in English.
- Code-switched: two languages inside a single segment, common with product, legal or technical terms.
- Thread-mixed: segments in different languages within one task while input and output share a language, such as a Spanish conversation carrying an internal note in English.
- Monolingual: every segment and the required output in one language.
The proxy agreement test: when can a geography field stand in for the text?
Only for stratifying a sample, and only where a labeled sample shows it agrees with the observed language. No proxy, however accurate, replaces measuring the text for the reported mix, because the cases where it disagrees are the ones the mix exists to find.
Run the test on each proxy separately. Compare the proxy value with the observed task class in a stratified sample and compute agreement for each value: the share of sampled tasks whose observed language matches the single language the proxy implies, with mixed and cross-lingual tasks counted as disagreement because no single-language field can predict them. Then read the result against this table:
| Proxy field | What it actually records | Why it diverges from the text | Allowed use if it agrees |
|---|---|---|---|
| Company headquarters or operating countries | Where the seller is based or has offices | Customers, suppliers and staff write in their own languages | Choosing which systems and regions to sample |
| Customer billing or account country | Where an account is billed or registered | Accounts move, multinationals buy centrally, and people write from abroad | Strata for the language sample |
| User locale or interface language | A profile setting, often a default | People rarely change defaults and write in whatever language the moment needs | Strata, after checking the share left at the default |
| Brand, queue or channel language | Routing rules set by the business | Overflow and escalation routes mix languages | Strata, plus a check on routing gaps |
| Region subtag on a language tag | A regional variety of a named language | It qualifies a language rather than naming one | Splitting varieties once the language is measured |
| One label per file or conversation | A single detector call over a whole unit | Mixed and cross-lingual tasks collapse into one label | None for the mix; segment labels are needed |
Sampling worksheet: draw tasks, label segments, record every proxy
Sample tasks, stratified by the proxies you want to test, and label language at the segment level inside each sampled task. The task is the unit because the buyer's model consumes tasks; Anthropic's engineering guide to agent evaluations likewise frames each task as one test with fixed inputs and pass criteria. A segment-level sample would overweight long threads and miss cross-lingual tasks altogether.
At a glance
Sampling worksheet: draw tasks, label segments, record every proxy
Language identification tools need checking on business text. fastText's published identification models recognize 176 languages and were trained on Wikipedia, Tatoeba and SETimes, which look little like ticket notes, part numbers or email signatures. An audit of web-crawled multilingual datasets by Kreutzer and colleagues found many corpora carrying incorrect or nontransparent language codes, and found non-linguistic content to be a more common problem than wrong-language content. Expect a version of both in business exports, where templates, codes and signatures receive confident labels that mean little.
Record these lines for every sampled task:
- Frame and unit: task ID, source system, frozen frame and draw seed, so another team can redraw the same sample.
- Proxies: every geography and locale field the dataset carries, recorded as found, including defaults and blanks.
- Segments: each message, note or field with its detected language, detector confidence and character count.
- Human check: a reviewer's label for short, low-confidence or language-light segments, with a second reviewer on a blind subset.
- Task reading: input languages, required output language and the task class from the precedence order.
- Results: task-class counts per stratum with 95 percent intervals, and agreement for each proxy value.
Worked example (illustrative): one support export, three account countries
Illustrative: a buyer evaluating a support export from a company with accounts in the United States, Canada and Mexico samples 300 tasks stratified by account country, then labels every segment. The seller's description said English for the U.S., English and French for Canada, and Spanish for Mexico.
Account country does predict something. For U.S. accounts, 96 of 120 sampled tasks were English only, 80 percent agreement. For Mexican accounts, 49 of 90 were Spanish only, about 54 percent, while 18 were English only. For Canada the proxy implies two languages, so it cannot be scored as a single-language predictor at all; its 27 other-language tasks were French.
The column a multilingual buyer cares about is the last one. Mixed and cross-lingual tasks appear under every country, from 10 of 120 U.S. tasks to 23 of 90 Mexican ones, and no country value identifies them. With 90 to 120 tasks per stratum, each share carries a 95 percent margin of up to about 10 points, enough to see that the Mexican mixed share, about 26 percent, sits well above the U.S. figure of about 8 percent, but not enough to rank smaller differences. All figures are invented to show the method; no measured language-mix results from SourceX are published here or elsewhere yet.
| Account country | Sampled tasks | One language: English | One language: other | Mixed or cross-lingual |
|---|---|---|---|---|
| United States | 120 | 96 | 14 | 10 |
| Canada | 90 | 51 | 27 | 12 |
| Mexico | 90 | 18 | 49 | 23 |
| All sampled tasks | 300 | 165 | 90 | 45 |
Can volume make the country label good enough?
No. A larger dataset narrows the interval around a proxy's agreement rate, but it does not change what the proxy measures, and it can make the mix look more settled than it is.
Large counts hide three problems. Repeated content, such as auto-replies, macros and signatures, inflates the dominant language without adding tasks. A few large accounts can supply most records in a country stratum, so the stratum's mix is really those accounts' mix. And tasks that failed or moved to another channel, such as a Spanish-speaking caller transferred to a phone line, never reach the text export, so a mix measured on records cannot show language demand that went unserved.
Two checks keep volume honest: deduplicate templates and repeated messages before labeling, and report the mix with the largest accounts removed as well as included. If the mix changes materially, the dataset's language profile belongs to its biggest customers, and the description should say so.
Does the observed mix match the mix the buyer will deploy into?
Compare the two by task class, because a dataset can match a deployment's language shares and still miss its cross-lingual work. Start from the buyer's target mix, expressed in the same classes, and read each gap as one of four decisions.
At a glance
Does the observed mix match the mix the buyer will deploy into
A language mix describes a sample, not a deployment. State the frame, date range, unit and intervals, and say which task classes the sample was too small to estimate. Write any gap that remains into the acceptance test rather than leaving it for a later surprise. The four decisions are:
- Accept: every task class the deployment needs is present within its interval, with enough examples to train or evaluate on.
- Rebalance: the classes exist in the wrong proportions, so draw a stratified subset that matches the target.
- Supplement: a needed class, often cross-lingual work, is too rare to estimate, so ask for records from the channels or regions where it occurs.
- Narrow the claim: describe the dataset as supporting the classes it holds, and drop the ones it does not.
How language mix enters a SourceX data request
SourceX judges language on the records, not on the seller's address. For companies outside the U.S., Canada and the EU, eligibility turns on the dataset being offered rather than the company's country or working language: the English share is measured on the actual records put forward, and each dataset still passes through rights, privacy, quality and buyer-fit review. Sellers in several listed countries need offered data that is at least 70 percent English, confirmed on the dataset itself.
Buyers state language needs in the request. SourceX does not hold datasets in stock; it finds them at U.S. businesses after a buyer asks, and the request-data form is where language sits beside scope, volume, format and timing. A request that names the target task classes, the input and output languages, and an acceptance test, such as a minimum count of cross-lingual tasks in a stated sample, gives a supplier something it can check against its own records.
Frequently asked questions
What is dataset language mix validation?
It checks that the languages a dataset is said to contain match the languages in its text, at the level of the tasks a model will see. It labels segments and tasks in a stratified sample, tests geography and locale fields against those labels, and reports the mix by task class with intervals.
Is a user's locale setting a reliable language label?
Not on its own. A locale such as en-US is a profile setting, often left at its default, and Zendesk's API documents it as a BCP 47 tag on the user record rather than on each message. Test it against labeled text before using it even to stratify a sample.
How should code-switched records be counted?
As their own task class rather than split between languages. A message that mixes Spanish and English needs a model that handles both at once, and assigning half of it to each language hides that requirement.
Can automatic language detection measure the mix by itself?
Use it for a first pass, then check it. Published models such as fastText's 176-language identifiers were trained on general text, and short, coded or templated business segments are where they slip. Have people label low-confidence and language-light segments, with a second reviewer on a blind subset.
Does a dataset need the same language mix as the deployment?
It needs every task class the deployment needs, in numbers large enough to train or evaluate on, not identical proportions. Proportions can be fixed by stratified sampling; a missing class, such as cross-lingual handoffs, can only be fixed by sourcing more records.
Sources
- In the 2017-2021 American Community Survey, more than 1 in 5 people (22 percent) aged 5 and older in the United States spoke a language other than English at home (Census Bureau, June 2025). Source
- RFC 5646 (BCP 47) region subtags indicate regional variation of a language, as in de-AT and es-419. Source
- Zendesk's Users API documents a user's locale as a BCP 47 tag, such as en-US, on the user record. Source
- fastText's language identification models recognize 176 languages and were trained on Wikipedia, Tatoeba and SETimes. Source
- Kreutzer et al. audited 205 language-specific corpora and found incorrect or nontransparent language codes, with non-linguistic content more common than wrong-language content. Source
- LinCE combines ten corpora across four code-switched language pairs and evaluates tasks including language identification and named entity recognition. Source
- Anthropic's engineering guide to agent evals describes a task as a single test with defined inputs and success criteria. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.