Privacy and preparation
Support multilingual handoff scope: what translated summaries drop
By SourceX Editorial · Updated
9 sections
Short answer
To test whether a translated support summary drops decision-critical details, list the facts that decided the case from the original-language messages, then score the summary fact by fact as present, altered, missing or added. Locate each loss in translation or summarization, link it to what the next agent did, and keep the originals with every case you include.
Countries and languages
Support multilingual handoff
Key takeaways
- Score translated summaries against the original-language record, never against another translation.
- Extract decision-critical facts first: negations, conditions, identifiers, deadlines, commitments, policy versions and hedges.
- Separate translation loss from summarization loss; they fail differently and need different fixes.
- Automatic MT metrics screen translations but cannot tell whether a short summary kept the deciding fact.
- Include only complete, matured cases and keep the originals; lossy summaries become labeled failure cases, not gold summaries.
How do you test whether translation summaries omit decision-critical details?
Test a translated summary against the original-language record, not against another translation. For each case, a bilingual reviewer lists the facts that decided the outcome from the original messages, then checks the summary fact by fact: present and correct, present but altered, missing, or added without support. Weight the facts by how much each one changes the decision, and compare failures with what happened next in the case.
This matters whenever one agent hands a case to another across a language boundary: a tier one team working in Spanish escalates to an English-speaking engineering group, or an outsourced team summarizes a call for an in-house specialist. The receiving agent acts on the summary, so a dropped negation or deadline can decide the case as surely as a wrong answer.
It also matters for AI. A support case used to train or evaluate a model is only as good as its record, and a record built from translated summaries can carry the same omissions into the model. The test below tells you which cases are safe to include, which need the originals attached, and which to leave out.
Two transformations, two ways to lose a fact
A translated handoff summary goes through two separate transformations, translation and summarization, and each fails in its own way. Measuring them together hides which step lost the detail.
Translation errors are well catalogued. The MQM error typology, used widely in translation quality work, places addition, omission and mistranslation under its Accuracy dimension: content that was not in the source, content from the source that is missing, and meaning that changed. Summarization adds a different risk. Maynez and colleagues found in a large human evaluation that neural abstractive summarizers are highly prone to produce content that is not faithful to the input document.
Support tools increasingly do both automatically. Zendesk's conversation translation, for example, shows agents translated end-user messages with a control to reveal the original text, and the ticket event log keeps end-user comments in their original language. That design is helpful for this test, because the source text survives alongside the translation.
Direction matters too. A summary written in English from a customer's Portuguese messages, and an English summary later translated back for a local agent, can lose different things. Note the direction of every step in the chain, because the same fact can survive the first hop and disappear on the second.
Which details decide a support case?
A decision-critical detail is any fact that, if dropped or changed, would lead a reasonable agent to take a different action. Most of them are short, which is why fluent summaries lose them without looking wrong.
Build the list from the original-language messages, the case fields and the outcome, before reading the summary, so the summary cannot shape what you look for. The examples in the table are illustrative.
| Detail type | Illustrative example | Why it decides the case | Typical failure |
|---|---|---|---|
| Negation and exceptions | The customer did not install the update | Changes the diagnosis path | Negation dropped in translation |
| Conditions | Replace only if the serial is in the affected range | Sets eligibility | Condition summarized away |
| Quantities and identifiers | Three units, order and serial numbers | Determines scope and the record to act on | Digits transposed or omitted |
| Dates and deadlines | Needs it before the site opens on the 14th | Sets priority and shipping method | Relative dates resolved wrongly |
| Commitments already made | The agent promised a call back by Friday | Binds the next agent | Promise missing from the summary |
| Policy or version cited | Applies the warranty policy version shown to the customer | Determines the allowed remedy | Generic policy substituted |
| Customer intent and constraints | Prefers email; cannot take calls at work | Sets channel and tone | Preference dropped as minor |
| Uncertainty | The customer thinks, but is not sure, it was dropped | Affects liability and next steps | Hedge turned into a statement |
The translation fidelity and evidence retention test
The test is a fixed procedure that two reviewers can run on the same case and reach the same scores. Run it on a sample before any multilingual support records go into a dataset, and keep the scoring sheet with each case.
Calibrate before scaling. Have both reviewers score the same small pilot set, compare their fact lists and scores, and rewrite the detail types or criticality definitions wherever they disagree. Only then split the remaining cases between reviewers, keeping a share double-scored throughout so drift shows up early.
- 1. Freeze the record: export the full ticket history with original-language messages, field changes and the handoff summary exactly as the receiving agent saw it.
- 2. Extract facts: a reviewer fluent in the source language lists every decision-critical fact from the originals, each with a type from the table above and a criticality of high, medium or low.
- 3. Score the summary: a second reviewer marks each fact present, altered, missing or contradicted, and lists any unsupported additions.
- 4. Locate the loss: for each failure, check the machine or human translation of the source message to decide whether the fact was lost in translation, in summarization, or was never stated clearly.
- 5. Link to outcome: record whether the receiving agent acted on the lost fact, asked the customer again, reopened the case or produced a wrong resolution.
- 6. Decide inclusion: apply the decision table below and record the reason.
| Result | Inclusion decision | What ships with the case |
|---|---|---|
| All high-criticality facts present and correct | Include | Summary, originals and scoring sheet |
| A high-criticality fact altered or missing, outcome unaffected | Include with flag | Originals required; summary marked as lossy |
| A high-criticality fact lost and outcome affected | Include only as a labeled failure case, or exclude | Loss location and outcome link |
| Unsupported additions in the summary | Flag; exclude from training as a gold summary | The added statements listed |
| Originals unavailable | Exclude | Exclusion reason only |
Can automatic translation metrics do this job?
Automatic metrics can screen translations, but none of them answers whether a summary kept the fact that mattered. Use them to decide where to spend reviewer time, not as the test.
Surface metrics such as BLEU, chrF and TER compare a translation with a reference by overlapping characters or words; SacreBLEU computes them reproducibly and prints a signature of the settings used. Neural metrics such as COMET score translation quality from 0 to 1, with or without a reference, and its XCOMET models mark error spans as minor, major or critical using the MQM categories.
The gap is structural. These tools score a translation against its source sentence or a reference translation. A handoff summary is shorter by design and has no reference, so a correct summary that leaves out small talk and an incorrect one that leaves out a negation can score alike. A reference-free quality score on the translation step, followed by human fact checking on the summary, is a reasonable division of labor.
Setting the support multilingual handoff scope
Scope the set to complete cases in which a handoff crossed a language boundary and the outcome had time to mature. A translated summary is only worth testing when you can see what the next agent did with it and whether the customer came back.
Include cases where the original messages, the summary and the downstream actions all survive; cases with a correction or reopening, because they show the cost of a lost detail; and a spread of language pairs, channels and products. Exclude cases still open, cases whose originals were deleted under a retention rule, and cases where the language boundary was crossed only by a customer who wrote in two languages without any handoff.
Record how each translation was produced: built-in machine translation in the help desk, an external translation tool, a bilingual agent writing the summary directly, or an outsourced team. The failure patterns differ, and a buyer evaluating a model needs to know which kind of handoff the cases represent. When the tool or its settings changed during the sampling period, treat the periods as separate strata.
Set an outcome maturity window before sampling, such as a fixed number of days after closure with no reopening or repeat contact, and apply it to every case. Without it, recent cases look resolved simply because the customer has not had time to come back.
Illustrative case: a replacement request that lost its condition
The customer and company in this case are made up to show how the test runs. A customer writes in German that two of five sensors stopped reporting, that they did not apply the latest firmware, and that a replacement is acceptable only if it arrives before an inspection date.
The tier one agent's English handoff summary reads that the customer has sensors offline and wants replacements. The fact extraction lists six decision-critical facts. Scoring finds three present, one altered (five units instead of two), and two missing: the firmware negation and the delivery condition.
Checking the translation step shows the machine translation kept all six facts, so the loss happened in summarization. The receiving engineer pushed a firmware update first, the customer replied that this was the wrong step, and the case reopened once. Under the decision table, the case is included only as a labeled failure, with originals attached and the loss located in the summary.
Reopenings, repeat contacts and the maturity window
Closure alone does not prove resolution. Treat a case as resolved only when the maturity window passes with no reopening, no repeat contact about the same issue through any channel, and, where relevant, a completed downstream action such as a shipped replacement or a deployed fix.
Linking those signals requires a stable customer or account key across channels, and the ticket history itself. Zendesk's Ticket Audits API, for instance, returns a read-only history of updates to a ticket, including comments and field changes with previous and new values, which is enough to see a status move from solved back to open. Repeat contacts on a different ticket need a link rule written down in advance, such as same account, same product and a similar issue within the window.
| Signal | Where it comes from | What it shows |
|---|---|---|
| Reopening | Ticket status history | The first resolution did not hold |
| Repeat contact on the same issue | Linked tickets by account and product | The problem returned or was not understood |
| Downstream action completed | Order, RMA or engineering records | The promised remedy actually happened |
| Satisfaction or follow-up reply | Survey and reply events | The customer's own view, where captured |
Original-language evidence, privacy and licensing
Keep the original-language messages with every case; a summary or translation should never be the only copy of what the customer said. That is also where privacy work concentrates, because names, addresses and identifiers appear in both languages, and redaction rules tuned for English can miss them in the original. A privacy reviewer should check de-identification on both versions.
Separate four questions before any records leave the company: whether you can export them, whether privacy law and your customer terms allow the intended use, who owns the content, and whether you hold the right to license it. De-identification does not answer the licensing question on its own.
For a support organization that goes ahead, SourceX runs the rights review, contracting, preparation and delivery of the case histories it licenses to AI developers. It works with companies in the U.S., Canada and the EU whose headcount peaked at 50 or more full-time employees, not counting contractors, and considers companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia only when at least 90% of the offered data is in English. That share is measured on the records actually offered, so a multilingual support archive needs the measurement before anything else. Before any of that, a short assessment, done without uploading files, shows whether the company qualifies.
Frequently asked questions
What should a translation fidelity and evidence retention test include?
It needs a frozen copy of the original-language record, a list of decision-critical facts extracted before reading the summary, a fact-by-fact score of the summary, a check of where each loss happened, a link to the case outcome and an inclusion decision. Two reviewers should be able to run it on the same case and agree.
Is a high machine translation score enough to trust a handoff summary?
No. Metrics such as BLEU, chrF and COMET score a translation against its source or a reference, not whether a much shorter summary kept the facts that drove the decision. Use them to screen translations, then check the summary by hand.
Who should review multilingual support cases?
A reviewer fluent in the source language extracts the facts, and a second reviewer scores the summary. A support lead who knows the product and policies should settle disagreements about which facts were decision-critical.
Should cases with lossy summaries be removed from a dataset?
Not always. When the original messages are kept and the loss is labeled, a lossy summary makes a useful failure example for evaluation. It should not be used as a gold summary for training.
How long should you wait before calling a case resolved?
Set a fixed outcome maturity window before sampling and apply it to every case. A case counts as resolved only when that window passes with no reopening and no repeat contact about the same issue.
Sources
- SacreBLEU computes shareable BLEU, chrF, chrF++ and TER scores and prints a signature recording the settings used, so scores can be reproduced. Source
- COMET provides reference-based and reference-free translation quality models scored from 0 to 1, and its XCOMET models flag error spans as minor, major or critical under the MQM typology. Source
- The MQM error typology groups translation errors into high-level dimensions; its Accuracy dimension includes addition, omission and mistranslation. Source
- Maynez et al. (ACL 2020) report from a large human evaluation that neural abstractive summarization models are highly prone to generate content unfaithful to the input document. Source
- Zendesk's conversation translation in the Agent Workspace shows translated end-user messages with a Show original control that reveals the untranslated text. Source
- Zendesk explains that the ticket event log keeps end-user comments in their original language and shows both the original and translated versions of agent messages. Source
- Zendesk's Ticket Audits API is a read-only history of all updates to a ticket, where each audit can contain events such as comments and field changes with previous and new values. Source
See if your company qualifies
A short company assessment. No data uploads are needed.