Skip to content

Privacy and preparation

English dataset mixed language exclusions: the workflow steps you lose

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Excluding non-English records from an English dataset can delete whole workflow steps, because language often follows the stage of the work: a plant's investigation in Spanish, a supplier's quote in German. Before excluding, tag language per segment, map every record to its stage, and compare stage coverage, thread breakage and outcome survival before and after the cut. Then exclude by thread or scope, not record, where stages would collapse.

Countries and languages

English dataset mixed

This page covers english dataset mixed language exclusions: the workflow steps you lose.Illustrative

Key takeaways

  • The row count falls by the non-English share, but the share of complete workflows can fall much further when language tracks a stage.
  • Tag language per segment with standard BCP 47 tags and a pinned detector, and review short or low-confidence segments by hand.
  • Measure stage coverage, thread breakage and outcome survival before and after the exclusion, and write the rule down before looking.
  • Exclude whole threads or a whole scope when record-level cuts would leave outcomes without causes.
  • English summaries and translations of excluded originals are derived text and must be labeled and linked, not passed off as originals.

Why does excluding non-English records remove workflow steps, not just rows?

Because in most international businesses, language follows the stage of the work. The customer writes in English, the overseas plant investigates in Spanish, the supplier answers in German, and the account manager closes the case in English. Remove every non-English record and the case still opens and closes, but the investigation in the middle is gone.

That changes what the dataset can teach. A model trained on the English-only remainder sees complaints followed by resolutions with no diagnosis between them, or purchase requests followed by approvals with no supplier negotiation. The row count drops by the non-English share; the share of complete workflows can drop by far more.

The fix is not to keep everything, since an English-language dataset may need a high English share for good reasons. It is to measure what an exclusion removes at the level of stages and threads before deciding how to exclude, and to write the result down.

Where language tends to follow the stage of the work

The patterns below are illustrative rather than measured; they show where to look first. Any workflow that crosses a border, a subsidiary or a bilingual crew is a candidate.

Internal stages are the most exposed. Customer-facing messages in an international business are often written in English by policy, while the notes, investigations and supplier exchanges behind them are written in whatever language the people doing the work share. Those internal records usually carry the reasoning a buyer of operational data cares about most.

Where language tends to follow the stage of the work
WorkflowStage often written in another languageWhat an English-only cut leavesSignal to check
Customer returns with an overseas plantRoot-cause investigation and corrective action at the plantComplaint and credit note with no diagnosisThreads where the investigation stage has zero English records
Cross-border purchasingSupplier quotes, counteroffers and technical clarificationsRequest and purchase order with no negotiationShare of purchase orders whose quote documents are excluded
Field service with bilingual crewsTechnician notes and photo captions on siteDispatch and invoice with no work descriptionJobs whose only note is excluded
Customs and freightCarrier, broker and customs documents from the origin countryShipment booking and delivery with no clearance stepShipments missing every origin-side document
Support escalated to a regional teamInternal escalation notes and engineering repliesFirst reply and closure with no escalationTickets with internal notes only in the excluded language
Intercompany financeSubsidiary invoices, approvals and dispute emailsLedger entries with no supporting disputeDisputes whose supporting records are all excluded

How should language be tagged before anything is excluded?

Tag language per segment, not only per file or record, and keep the original next to any translation. A ticket can hold an English customer message, a Spanish internal note and a German attachment; a single record-level label hides two of the three.

Use standard tags. The W3C's guide to language tags explains that tag syntax comes from BCP 47, currently RFC 5646, and recommends keeping tags as short as possible, adding a region or script subtag only when it adds useful information, as in es-419 for Spanish used in Latin America or zh-Hans for Chinese in simplified script. Region matters here more than usual when it changes date, number or unit conventions.

Choose a detector whose output supports segments and uncertainty. Google's CLD3 can split text into spans by script and return, for each predicted language, a probability, a reliability flag, the proportion of bytes in that language and the byte ranges it covers. The Lingua library documents detection of multiple languages in mixed text but describes it as experimental, with lower accuracy on short phrases. fastText's published models recognize 176 languages, and a newer NLLB model uses ISO 639-3 codes with a script suffix. Whatever you pick, pin the version and record it with the tags.

Expect labels to be wrong sometimes. An audit of 205 language-specific corpora from five public web-mined datasets by Kreutzer and colleagues found that many were mislabeled or used nonstandard or ambiguous language codes. Business text is cleaner than web crawl, but short notes, product codes and names still confuse detectors, so sample and review the segments the exclusion depends on.

How do you map records to workflow stages when systems do not label them?

Build the stage map from fields the systems already keep, and let people fill only the gaps. Most operational systems record enough to infer a stage: the record type (quote, purchase order, inspection, credit note), the status at the time of writing, the queue or team that owned it, and the author's role. A short mapping table from those values to a fixed stage list per workflow does most of the work.

Keep the stage list small, usually five to eight stages per workflow, and include one stage for anything the map cannot place. Name stages for what happens, such as intake, diagnosis, negotiation, decision and closure, rather than for the system that holds them, so the same list works across tools and subsidiaries.

Then check the map against the language tags. Cross-tabulate stage by language for each workflow. Where one stage is mostly in one language and the others are mostly in English, you have found the place where an exclusion will bite, and the audit steps that follow will show by how much.

Thread identity matters as much as stage. Use the system's own case, order, job or ticket ID to group records, and link attachments and emails to it through the IDs they carry. Where the link is only a subject line or a date, record it as inferred, because a wrong thread assignment will make breakage look smaller or larger than it is.

The language exclusion impact and completeness audit

The audit is designed for the person who will defend the dataset later, whether to a buyer, an auditor or a licensing review. Every measure compares the same scope before and after the cut, so the difference is attributable to the exclusion and nothing else.

Run the audit on the full candidate set before any exclusion, then again on the proposed remainder, and compare. Steps one and two prepare the data; steps three to eight measure the impact; step nine records the decision. The thresholds in the red-flag column are editorial rules of thumb, not standards; set your own and write them down before you look at the results.

The language exclusion impact and completeness audit
StepMeasureHow to compute itRed flag (editorial rule of thumb)
1. Tag segmentsLanguage tag, detector version and reliability per segmentRun the pinned detector; human-review a sample of low-reliability and short segmentsUnreviewed low-reliability segments decide whether a record is excluded
2. Map stagesWorkflow stage for every record and segmentMap record types, statuses, authors or queues to a fixed stage list per workflowMore than a small share of records cannot be assigned a stage
3. Stage coverageShare of threads in which each stage is present, before and afterPer stage: threads with at least one surviving record divided by threads in scopeAny stage loses far more coverage than the overall row loss
4. Thread breakageShare of threads that lose at least one recordCompare record lists per thread ID before and afterBroken threads outnumber whole threads removed
5. Outcome survivalShare of threads whose closing or outcome record survives without its causeThreads with outcome present and cause stage absentOutcomes survive while causes vanish
6. Stratum skewChange in share by entity, site, region, product or customer segmentDistribution before and after, per stratumA whole site, subsidiary or product line falls below a usable count
7. Locale carry-overSurviving records that rely on excluded ones for dates, units, amounts or referencesFind references to excluded IDs; flag ambiguous dates and decimal separatorsSurviving records cite figures only an excluded record explains
8. Derived textEnglish summaries or translations of excluded originalsFind records marked or detected as translations or summariesDerived English stands in for an excluded original with no link to it
9. Decision logChosen rule, counts, measures 3 to 8 and the reviewerOne page per workflow, stored with the datasetNo written rule, or a rule changed after seeing results

Four assumptions that can make an acceptable exclusion misleading

Four assumptions most often turn an acceptable exclusion into a misleading one, and each can be tested before the decision.

  • That English records are self-contained: a surviving English message that says the plant confirmed the cause is meaningless if the confirmation was the excluded record.
  • That dates and numbers read the same everywhere: 03/04 is March or April depending on locale, and a decimal comma can turn a tolerance into a different value. Keep the source locale on every surviving record that came from a non-English context.
  • That language marks jurisdiction: an English email sent by a foreign subsidiary may still be governed by that country's law and contracts, so excluding by language does not exclude by jurisdiction. Tag entity and country separately.
  • That the detector is right on short text: product codes, part numbers, names and one-line notes are where detectors are least reliable, and they are exactly the segments that decide many exclusions.

Worked example (illustrative): a returns workflow shared with a plant in Mexico

This example is illustrative; the company, counts and percentages are invented to show how the audit reads. A U.S. distributor wants to offer six years of customer returns records as an English dataset. Of the candidate records, about 7% are in Spanish, nearly all from its plant in Mexico.

Step three changes the picture. Dropping the Spanish records removes 7% of rows, but the root-cause stage falls from present in 88% of returns threads to 12%, because the plant's quality team wrote its investigations in Spanish. Step four shows about 31% of threads lose at least one record, and step five shows that in most of them the credit note survives while its cause does not.

Step eight finds an exception. For one product line, account managers wrote English summaries of the plant's findings into the ticket. Those threads still show the investigation, but through derived text, so they are kept and labeled as summaries with a link to the excluded originals rather than presented as original English investigation records.

Step six adds a second warning. Nearly all of the plant's own records sit in the excluded share, so the English remainder says almost nothing about one of the company's three production sites. Any claim that the dataset covers the whole returns operation would be wrong, and the decision log says so plainly.

The decision log records three options and the choice. Excluding record by record was rejected because it leaves broken threads. Keeping the Spanish records was possible only if the offered scope's English share stayed within what the buyer or licensing review accepts. The distributor chose to exclude whole threads that depended on the plant and to offer the remaining complete threads as a narrower set, with the excluded share and the reason written down.

Exclude by record, by thread or by scope?

Exclude by the unit the dataset will be used in. If a buyer trains on whole cases, exclude whole threads; if it trains on single messages, record-level exclusion may be fine, provided the audit shows no stage collapses.

Whichever unit you choose, report two numbers side by side: the share of rows removed and the share of threads removed or broken. A buyer reading only the first will overestimate what remains. If the two numbers are far apart, that gap is the main finding of the audit and belongs in the dataset's documentation, not only in the internal decision log.

  • Record level: smallest loss of rows, highest risk of broken threads and orphaned outcomes. Use only when records stand alone.
  • Thread level: removes every thread that depends on an excluded record. Fewer rows, but every remaining thread is complete and honest, and the count of removed threads is easy for a reviewer to check against the decision log.
  • Scope level: drop a workflow, site or subsidiary where language and stage are tied, and offer the rest. Simplest to explain and to verify, and it keeps the description of the dataset accurate about which operations it covers.
  • Keep and label: retain non-English segments within a stated share, tagged and with originals linked to any translation. Suits buyers who want realistic mixed records.
  • Pair translations: add a translation next to each original, labeled as derived. It restores readability but not originality, and the translation is a new artifact with its own provenance.

Excluded records still need rights and privacy decisions

An excluded record is not a deleted one, and the decision to exclude it does not settle anything about the records that remain. Translations and English summaries of excluded originals carry the same personal data and confidential content as the originals. Names and identifiers in non-English text are easier to miss, because detection tools are often tuned for English, so check the surviving derived text as carefully as the originals.

The excluded records also stay in your systems and in your retention schedule. Note in the decision log where they live, so that a later request to add them back, or a deletion request from a person named in them, can be handled for both the excluded and the offered parts.

Records created by a foreign subsidiary or exchanged with foreign counterparties can be subject to that country's privacy and contract rules even when written in English. Tag the entity and the country of origin, and have counsel review cross-border scope for the specific records before anything is offered.

How the English share is judged on the records you actually offer

For licensing, the English share is a property of the offered scope, not of the company. SourceX works with U.S., Canadian and EU companies that had 50 or more full-time employees at their peak (contractors excluded), and companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia can be considered when at least 90% of the offered data is in English. That English share is measured on the records actually offered, and the set still goes through rights, privacy, quality and buyer-fit review. The audit above helps a company reach that share without hollowing out its workflows.

The first step with SourceX is a short company assessment that needs no data upload, so the exclusion decision can be shaped before any records move.

Frequently asked questions

Should non-English records be excluded record by record or thread by thread?

Thread by thread when the dataset will be used as whole cases, because record-level exclusion leaves outcomes without causes. Record by record only works when each record stands alone. Run the stage coverage and thread breakage measures before choosing.

Is a machine translation of a non-English record an English record?

Not for provenance purposes. It is a derived artifact that should be labeled as a translation, linked to its original and counted separately. Presenting it as original English text misstates what the dataset contains.

How reliable is automatic language identification on short business text?

Less reliable than on long text. The Lingua documentation says mixed-language detection is experimental and less accurate on short phrases, and an audit of web-mined corpora found many mislabeled language codes. Review a sample of short and low-confidence segments by hand.

What language tag format should a dataset use?

Use BCP 47 tags, defined in RFC 5646, and keep them as short as possible. Add a region or script subtag only when it adds information, for example when locale changes date and number conventions.

Does excluding records change what a model can learn from the dataset?

Yes, when language tracks a workflow stage. The remainder can teach that cases go from complaint to resolution with nothing in between. Measuring stage coverage before and after shows whether that has happened.

Sources

  • The W3C guide to language tags says tag syntax is defined by BCP 47, currently RFC 5646, recommends keeping tags as short as possible, and gives examples such as es-419 and zh-Hans. Source
  • Google's CLD3 can split text into script-based spans and return, for each predicted language, a probability, a reliability flag, the proportion of bytes and the byte ranges. Source
  • The Lingua language detector documents detection of multiple languages in mixed-language text as experimental, with lower accuracy on shorter phrases. Source
  • fastText distributes language identification models that recognize 176 languages, and a newer NLLB model uses ISO 639-3 codes with a script code. Source
  • Kreutzer et al. audited 205 language-specific corpora from five public web-mined datasets and found many were mislabeled or used nonstandard or ambiguous language codes. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies