Privacy and preparation
English business dataset translation boundaries: keep source context
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
To preserve non-English decision context inside English business documents, split records into segments, tag each segment's language with a BCP 47 code, keep every original next to any translation with linked IDs and a marked authoritative version, and store dates, currencies and units both as written and normalized with their locale. Then flag records whose outcome depends on a non-English segment and review their origin and rights.
Countries and languages
English business dataset
Key takeaways
- English records often carry the deciding context in quoted replies, notes and attachments written in another language.
- Keep originals and translations linked by ID, record who translated, and mark which version the business relied on.
- Tag language per segment with BCP 47 codes and human-check short or mixed segments.
- Store dates, currencies, units and legal terms as written and normalized, with their locale or jurisdiction.
- Language does not decide transfer or licensing rights, but it is a useful signal to check where records came from.
Where does non-English decision context hide in English business records?
In an English business dataset, the non-English parts are rarely whole documents. They are quoted replies, attachments, scanned notes, local legal terms and supplier names, and they are often where the decision was actually made. A record can be mostly English and still turn on one sentence in another language.
That matters for any later use of the records. A model trained or evaluated on the English parts alone learns the outcome without its cause, and a reviewer measuring how much of a dataset is English can miss the segments that carry the most meaning.
Cross-border businesses produce this pattern constantly. A U.S. distributor's purchasing records quote suppliers abroad, a Canadian service firm's tickets carry French customer replies, and a manufacturer's English quality system holds operator notes from a plant in another country. None of these would be labeled multilingual in a system inventory, yet each holds evidence a translation-only or English-only extract would lose.
Start by knowing where to look. The table lists the places the audit later checks, with an example of each and what goes wrong when the segment is dropped or flattened into an English summary.
| Where it hides | Example | What is lost if dropped or summarized |
|---|---|---|
| Quoted reply chains | An English ticket thread quoting a supplier's reply in Portuguese | The supplier's reason for the delay that drove the decision |
| Attachments and scans | A handwritten inspection note in Spanish on an English work order | The defect description the rejection was based on |
| Local legal and tax terms | A French contract clause name kept in an English summary | The specific legal meaning the English gloss does not carry |
| Free-text fields | Agent notes switching languages mid-sentence | Negations and qualifiers that a machine translation can flip |
| Names and identifiers | Company, product and place names in their original script | The ability to match records across systems |
| Machine-translated summaries | An auto-translated English summary stored instead of the original | Any way to check the summary against what was said |
Why a translation must stay linked to its original
Keep the original text, keep the translation, and link them, because a translation is derived evidence and only the original can settle a disagreement about what was said. Record who or what produced each translation and which version the business relied on when it acted.
Translation standards already model this. In XLIFF 2.1, the OASIS exchange format used across the translation industry, each segment holds one source and at most one target, and the document declares the source language and target language with BCP 47 codes; text in another language inside a segment is marked inline. The W3C Internationalization Tag Set (ITS 2.0) adds a Translate flag that says whether content is meant to be translated and a Localization Note that carries guidance a translator must read first.
You do not need to convert records into XLIFF. Borrow the ideas: a stable ID for the original segment, a separate ID for each translation, the language of each, a do-not-translate flag for names, codes and legal terms, and a note field for context a later reader needs. Mark which version was authoritative for the decision, because sometimes the business acted on a translation that was wrong.
Those wrong translations are valuable in their own right. A record where a team misread a supplier's note and then corrected course shows a real failure and its repair, which is exactly the kind of case that evaluation sets for business agents lack. It survives only if the original and the translation were both kept.
Which date, unit, currency or jurisdiction assumption could reverse a record's meaning?
Formats that look universal are the most dangerous, because nobody thinks to check them. A short date, a currency symbol or a unit abbreviation can be read two ways, and the wrong reading can reverse a late-delivery finding, a price comparison or a safety limit.
Dates are the classic case. Unicode's CLDR project, a widely used source of locale data, documents that short date patterns order day, month and year differently by locale, and its LDML specification notes that even sub-locales differ: Spanish as used in Spain defaults to 24-hour time while Spanish as used in the United States uses 12-hour time. ISO 8601 exists to remove this ambiguity with a year-month-day form such as 2026-10-18.
Currencies have the same problem. The dollar sign is used for many currencies, so store amounts with an ISO 4217 code, the three-letter alphabetic code such as USD or CAD, alongside the original text. Do the same for units: keep the unit as written and the unit you normalized to, rather than overwriting one with the other.
| Assumption | Ambiguous example | How it can reverse the answer | What to keep |
|---|---|---|---|
| Date order | 03/04/2025 on a delivery note | A shipment reads as early instead of a month late | Original string, source locale and ISO 8601 value |
| Clock format | Times with no AM or PM from a 12-hour locale | Events appear in the wrong order | Original string, locale and UTC value |
| Currency | A dollar sign on a supplier quote | A price comparison uses the wrong currency | Original text and ISO 4217 code |
| Decimal separator | 1,250 in a European measurement | A tolerance reads as a thousand times larger | Original string, locale and parsed number |
| Units | Tons, gallons or degrees without a system | A load or temperature limit is misread | Unit as written and normalized unit |
| Jurisdiction | A legal term from one country's law in an English note | An obligation is read under the wrong law | Original term, country and a do-not-translate flag |
How should language be tagged: per document or per segment?
Tag language per segment, not per document. A document-level tag says the file is English and hides every quoted reply and note inside it; segment tags let you count, review and route the non-English parts on their own.
Use BCP 47 language tags, the IETF standard published as RFC 5646. A tag starts with a language subtag and can add a script and a region, so sr-Latn means Serbian in Latin script and es-419 means Spanish as used in Latin America and the Caribbean. The standard advises tags as precise as possible but no more specific than is justified, which fits records well: tag fr-CA only when you know the text is Canadian French.
Store how each tag was set. Automatic language identification is fast but unreliable on short segments, names and mixed-language sentences, so record whether a tag came from a detector, a human or the source system, and send short or mixed segments to a human sample.
Decide the counting unit before you report a language share. Counting documents overstates English, because one English header makes a file English. Counting characters or tokens per tagged segment gives a share that matches what a model would actually read, and it can be recomputed by anyone who has the segment table. State the unit next to every percentage you publish.
What a segment record looks like when the boundary is kept
A segment record is one row per piece of text, carrying enough fields that someone who reads only the English can see where another language was, what it said, and how much it mattered. The fields below are a starting set; most can be filled automatically, and the last two need a person.
At a glance
What a segment record looks like when the boundary is kept
Store these alongside the parent record rather than inside it, so the parent keeps its original layout and the boundary information can be added, corrected or removed without editing the source.
- segment_id and parent_record_id: stable keys, so a quoted reply can be traced back to the email or ticket it came from.
- position: where the segment sits, such as body, quoted reply, attachment page or field name.
- lang_tag and tag_method: the BCP 47 tag and whether a detector, a person or the source system set it.
- original_text: the text exactly as captured, never overwritten.
- translation_id, translation_text and translator: an optional linked translation and whether a person, a vendor or a machine produced it.
- authoritative: which version the business relied on when it acted, original or translation.
- do_not_translate: a flag for names, product codes, identifiers and legal terms.
- locale_values: dates, numbers, currencies and units found in the segment, each stored as written, with locale and normalized value.
- jurisdiction: the country whose law or regulator the segment refers to, if any.
- decision_dependency: whether the parent record's outcome depends on this segment, set by a reviewer.
- review_note: what the reviewer needed to know to judge the segment, in the spirit of a localization note.
The language boundary and context completeness audit
Run this audit on a sample of records from each source system before the dataset is described as English or translated. Each row says what passes, so two reviewers reach the same result.
Draw the sample by source system and record type, then add every record that a keyword or detector flags as mixed-language. Pair a reviewer who reads the second language with one who knows the business process: the first can say what a note means, the second can say whether the decision depended on it. Record both names by role in the audit file and rerun the audit whenever a new source system or region is added.
| Check | What to record | Pass condition | Evidence |
|---|---|---|---|
| 1. Segment inventory | Every text segment with an ID and its parent record | Quoted text, attachments and notes are separate segments | Segment count per record type |
| 2. Language tag | BCP 47 tag and how it was set | Every segment tagged; short and mixed segments human-checked | Tag distribution and review sample |
| 3. Original kept | Original text for every translated segment | No translation exists without its original | Original-to-translation ID map |
| 4. Translation provenance | Who or what translated, when, and which version was relied on | Authoritative version marked for every decision record | Provenance fields filled |
| 5. Do-not-translate items | Names, codes, legal terms and identifiers | Flagged items match the original exactly | Flag list and spot check |
| 6. Dates and times | Original string, source locale, ISO 8601 value | No date parsed without a known locale | Ambiguous-date count and resolution |
| 7. Money and units | Original text, ISO 4217 code, unit as written and normalized | No amount or measure overwritten | Conversion log |
| 8. Jurisdiction | Country of origin and governing law for legal or regulatory segments | Every such segment has a country | Jurisdiction field coverage |
| 9. Decision dependency | Whether the record's outcome depends on a non-English segment | Dependent records flagged for review | Flagged record list |
| 10. Rights and access | Origin of personal data, where records are stored and accessed from | Transfer and rights review done for flagged sources | Review reference per source |
Worked example (illustrative): a cross-border quality file that reads as English
This example is invented to show the audit; the company and figures are illustrative. A U.S. manufacturer with a plant in Mexico wants to describe its supplier quality archive as an English dataset. The quality system's forms, headers and dispositions are in English, so a first pass marks the whole archive English.
The segment inventory tells a different story. In a sample of 200 nonconformance records, about 30 percent include a Spanish operator note or a quoted supplier email, and in roughly one in ten records the disposition of reject or use-as-is follows directly from that note. Several records also store only a machine-translated English summary of the note, with the original discarded.
Dates fail next. The plant's paper travelers use day-month-year while the U.S. system uses month-day-year, and early in each month the two cannot be told apart from the string alone. The team resolves them from the record's creation timestamp where possible and flags the rest as ambiguous rather than guessing.
The outcome: Spanish segments are tagged es-MX and kept with their originals, the summaries are marked as derived and non-authoritative, the records that depend on a Spanish note are flagged, and the archive is described as English with tagged Spanish segments, with the measured share stated. That description is accurate, and a reviewer can check it.
The exception is the records where only the machine summary survives. Without an original there is nothing to check the summary against, so those records are kept for operational use but excluded from any dataset that relies on the decision reasoning.
When a foreign-language segment changes the rights or transfer review
A non-English segment often signals that the records came from, or describe people in, another country, and that can change the legal review even when the dataset is presented as English. Treat language as a prompt to check origin and access, not as a legal finding on its own.
If records include personal data from the EU, the GDPR's Chapter V governs transfers to third countries: Article 44 says a transfer may take place only if the Chapter's conditions are met, and the following articles set out adequacy decisions, appropriate safeguards and narrow derogations. Where the records are stored and from which countries they are accessed both matter, which is why the audit records them.
Language can also flag contract and rights issues. A supplier's foreign-language terms quoted in an email, or a local agent's documents, may carry their own confidentiality or ownership terms. Translating or de-identifying them does not give the company a right to license them; counsel reviews those sources on their specific facts.
Translation itself can be a processing step that needs review. Sending segments that contain personal data to an outside translation service or a hosted machine translation tool moves the data to another party and possibly another country, so check that step against the same transfer and contract rules before running it at scale.
How the English share is judged when records are offered for licensing
For records offered for licensing, the English share should be measured on the actual records, segment by segment, which is exactly what this audit produces. A label based on the company's working language or the language of the forms will usually overstate it.
Language matters to eligibility as well as to model quality. SourceX's suppliers are U.S., Canadian and EU companies that employed 50 or more full-time staff at their peak, with contractors left out of the count, and companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia can also be considered when at least 90 percent of the data they offer is in English, a share confirmed on the actual dataset. For them the English share is measured on the records being offered, not inferred from the company's country or working language, and every dataset still goes through rights, privacy, quality and buyer-fit review. A company can find out whether it qualifies through a short assessment that needs no data uploads.
Frequently asked questions
Should non-English segments be translated before a dataset is licensed?
Not by default. Keep the original, add a translation only if the use needs one, link the two by ID and mark which is authoritative. Replacing the original with a translation removes the only way to check what was actually said.
How do you measure how much of a business dataset is English?
Split records into segments, including quoted text, notes and attachments, tag each with a BCP 47 language tag, and count by characters or tokens rather than by document. Check short and mixed segments with a human sample, since automatic detection is weakest there.
Is a machine-translated summary good enough to keep?
Keep it as derived text, clearly marked, next to the original. On its own it is weak evidence, because translation can drop negations, qualifiers and legal meaning, and there is nothing to verify it against once the original is gone.
What date format should a cleaned dataset use?
Store an ISO 8601 value such as 2026-10-18 for analysis, but keep the original string and the source locale beside it. When the locale is unknown and the value is ambiguous, flag it instead of guessing.
Does an English dataset avoid cross-border data rules?
No. Transfer rules depend on where personal data comes from and where it goes, not on the language it is written in. Non-English segments are a useful signal to check origin, and counsel decides how the rules apply.
Sources
- XLIFF 2.1 requires a BCP 47 source language, holds one source and at most one target per segment, and expects text in other languages inside a segment to be marked inline. Source
- ITS 2.0 defines a Translate data category for whether content is intended for translation and a Localization Note data category for notes to localizers. Source
- CLDR guidance documents that short date patterns order day, month and year differently by locale. Source
- The LDML dates specification gives the example that Spanish as used in Spain uses 24-hour time formats while Spanish as used in the United States uses 12-hour formats. Source
- ISO describes ISO 8601 as the internationally accepted way to represent dates and times, using a year-month-day form to remove ambiguity. Source
- ISO 4217 defines three-letter alphabetic codes and three-digit numeric codes for currencies, such as USD for the US dollar. Source
- RFC 5646 (BCP 47) defines language tags built from a language subtag with optional script and region subtags, with examples such as sr-Latn and es-419. Source
- GDPR Article 44 allows transfers of personal data to third countries only if the conditions of Chapter V are met, with adequacy, safeguards and derogations set out in the following articles. Source
See if your company qualifies
A short company assessment. No data uploads are needed.