Skip to content

Privacy and preparation

Support multilingual handoff scope: what translated summaries drop

By SourceX Editorial · Updated

9 sections

Short answer

To test whether a translated support summary drops decision-critical details, list the facts that decided the case from the original-language messages, then score the summary fact by fact as present, altered, missing or added. Locate each loss in translation or summarization, link it to what the next agent did, and keep the originals with every case you include.

Countries and languages

Support multilingual handoff

This page covers support multilingual handoff scope: what translated summaries drop.Illustrative

Key takeaways

  • Score translated summaries against the original-language record, never against another translation.
  • Extract decision-critical facts first: negations, conditions, identifiers, deadlines, commitments, policy versions and hedges.
  • Separate translation loss from summarization loss; they fail differently and need different fixes.
  • Automatic MT metrics screen translations but cannot tell whether a short summary kept the deciding fact.
  • Include only complete, matured cases and keep the originals; lossy summaries become labeled failure cases, not gold summaries.

How do you test whether translation summaries omit decision-critical details?

Test a translated summary against the original-language record, not against another translation. For each case, a bilingual reviewer lists the facts that decided the outcome from the original messages, then checks the summary fact by fact: present and correct, present but altered, missing, or added without support. Weight the facts by how much each one changes the decision, and compare failures with what happened next in the case.

This matters whenever one agent hands a case to another across a language boundary: a tier one team working in Spanish escalates to an English-speaking engineering group, or an outsourced team summarizes a call for an in-house specialist. The receiving agent acts on the summary, so a dropped negation or deadline can decide the case as surely as a wrong answer.

It also matters for AI. A support case used to train or evaluate a model is only as good as its record, and a record built from translated summaries can carry the same omissions into the model. The test below tells you which cases are safe to include, which need the originals attached, and which to leave out.

Two transformations, two ways to lose a fact

A translated handoff summary goes through two separate transformations, translation and summarization, and each fails in its own way. Measuring them together hides which step lost the detail.

Translation errors are well catalogued. The MQM error typology, used widely in translation quality work, places addition, omission and mistranslation under its Accuracy dimension: content that was not in the source, content from the source that is missing, and meaning that changed. Summarization adds a different risk. Maynez and colleagues found in a large human evaluation that neural abstractive summarizers are highly prone to produce content that is not faithful to the input document.

Support tools increasingly do both automatically. Zendesk's conversation translation, for example, shows agents translated end-user messages with a control to reveal the original text, and the ticket event log keeps end-user comments in their original language. That design is helpful for this test, because the source text survives alongside the translation.

Direction matters too. A summary written in English from a customer's Portuguese messages, and an English summary later translated back for a local agent, can lose different things. Note the direction of every step in the chain, because the same fact can survive the first hop and disappear on the second.

Which details decide a support case?

A decision-critical detail is any fact that, if dropped or changed, would lead a reasonable agent to take a different action. Most of them are short, which is why fluent summaries lose them without looking wrong.

Build the list from the original-language messages, the case fields and the outcome, before reading the summary, so the summary cannot shape what you look for. The examples in the table are illustrative.

Which details decide a support case?
Detail typeIllustrative exampleWhy it decides the caseTypical failure
Negation and exceptionsThe customer did not install the updateChanges the diagnosis pathNegation dropped in translation
ConditionsReplace only if the serial is in the affected rangeSets eligibilityCondition summarized away
Quantities and identifiersThree units, order and serial numbersDetermines scope and the record to act onDigits transposed or omitted
Dates and deadlinesNeeds it before the site opens on the 14thSets priority and shipping methodRelative dates resolved wrongly
Commitments already madeThe agent promised a call back by FridayBinds the next agentPromise missing from the summary
Policy or version citedApplies the warranty policy version shown to the customerDetermines the allowed remedyGeneric policy substituted
Customer intent and constraintsPrefers email; cannot take calls at workSets channel and tonePreference dropped as minor
UncertaintyThe customer thinks, but is not sure, it was droppedAffects liability and next stepsHedge turned into a statement

The translation fidelity and evidence retention test

The test is a fixed procedure that two reviewers can run on the same case and reach the same scores. Run it on a sample before any multilingual support records go into a dataset, and keep the scoring sheet with each case.

Calibrate before scaling. Have both reviewers score the same small pilot set, compare their fact lists and scores, and rewrite the detail types or criticality definitions wherever they disagree. Only then split the remaining cases between reviewers, keeping a share double-scored throughout so drift shows up early.

  • 1. Freeze the record: export the full ticket history with original-language messages, field changes and the handoff summary exactly as the receiving agent saw it.
  • 2. Extract facts: a reviewer fluent in the source language lists every decision-critical fact from the originals, each with a type from the table above and a criticality of high, medium or low.
  • 3. Score the summary: a second reviewer marks each fact present, altered, missing or contradicted, and lists any unsupported additions.
  • 4. Locate the loss: for each failure, check the machine or human translation of the source message to decide whether the fact was lost in translation, in summarization, or was never stated clearly.
  • 5. Link to outcome: record whether the receiving agent acted on the lost fact, asked the customer again, reopened the case or produced a wrong resolution.
  • 6. Decide inclusion: apply the decision table below and record the reason.
The translation fidelity and evidence retention test
ResultInclusion decisionWhat ships with the case
All high-criticality facts present and correctIncludeSummary, originals and scoring sheet
A high-criticality fact altered or missing, outcome unaffectedInclude with flagOriginals required; summary marked as lossy
A high-criticality fact lost and outcome affectedInclude only as a labeled failure case, or excludeLoss location and outcome link
Unsupported additions in the summaryFlag; exclude from training as a gold summaryThe added statements listed
Originals unavailableExcludeExclusion reason only

Can automatic translation metrics do this job?

Automatic metrics can screen translations, but none of them answers whether a summary kept the fact that mattered. Use them to decide where to spend reviewer time, not as the test.

Surface metrics such as BLEU, chrF and TER compare a translation with a reference by overlapping characters or words; SacreBLEU computes them reproducibly and prints a signature of the settings used. Neural metrics such as COMET score translation quality from 0 to 1, with or without a reference, and its XCOMET models mark error spans as minor, major or critical using the MQM categories.

The gap is structural. These tools score a translation against its source sentence or a reference translation. A handoff summary is shorter by design and has no reference, so a correct summary that leaves out small talk and an incorrect one that leaves out a negation can score alike. A reference-free quality score on the translation step, followed by human fact checking on the summary, is a reasonable division of labor.

Setting the support multilingual handoff scope

Scope the set to complete cases in which a handoff crossed a language boundary and the outcome had time to mature. A translated summary is only worth testing when you can see what the next agent did with it and whether the customer came back.

Include cases where the original messages, the summary and the downstream actions all survive; cases with a correction or reopening, because they show the cost of a lost detail; and a spread of language pairs, channels and products. Exclude cases still open, cases whose originals were deleted under a retention rule, and cases where the language boundary was crossed only by a customer who wrote in two languages without any handoff.

Record how each translation was produced: built-in machine translation in the help desk, an external translation tool, a bilingual agent writing the summary directly, or an outsourced team. The failure patterns differ, and a buyer evaluating a model needs to know which kind of handoff the cases represent. When the tool or its settings changed during the sampling period, treat the periods as separate strata.

Set an outcome maturity window before sampling, such as a fixed number of days after closure with no reopening or repeat contact, and apply it to every case. Without it, recent cases look resolved simply because the customer has not had time to come back.

Illustrative case: a replacement request that lost its condition

The customer and company in this case are made up to show how the test runs. A customer writes in German that two of five sensors stopped reporting, that they did not apply the latest firmware, and that a replacement is acceptable only if it arrives before an inspection date.

The tier one agent's English handoff summary reads that the customer has sensors offline and wants replacements. The fact extraction lists six decision-critical facts. Scoring finds three present, one altered (five units instead of two), and two missing: the firmware negation and the delivery condition.

Checking the translation step shows the machine translation kept all six facts, so the loss happened in summarization. The receiving engineer pushed a firmware update first, the customer replied that this was the wrong step, and the case reopened once. Under the decision table, the case is included only as a labeled failure, with originals attached and the loss located in the summary.

Reopenings, repeat contacts and the maturity window

Closure alone does not prove resolution. Treat a case as resolved only when the maturity window passes with no reopening, no repeat contact about the same issue through any channel, and, where relevant, a completed downstream action such as a shipped replacement or a deployed fix.

Linking those signals requires a stable customer or account key across channels, and the ticket history itself. Zendesk's Ticket Audits API, for instance, returns a read-only history of updates to a ticket, including comments and field changes with previous and new values, which is enough to see a status move from solved back to open. Repeat contacts on a different ticket need a link rule written down in advance, such as same account, same product and a similar issue within the window.

Reopenings, repeat contacts and the maturity window
SignalWhere it comes fromWhat it shows
ReopeningTicket status historyThe first resolution did not hold
Repeat contact on the same issueLinked tickets by account and productThe problem returned or was not understood
Downstream action completedOrder, RMA or engineering recordsThe promised remedy actually happened
Satisfaction or follow-up replySurvey and reply eventsThe customer's own view, where captured

Original-language evidence, privacy and licensing

Keep the original-language messages with every case; a summary or translation should never be the only copy of what the customer said. That is also where privacy work concentrates, because names, addresses and identifiers appear in both languages, and redaction rules tuned for English can miss them in the original. A privacy reviewer should check de-identification on both versions.

Separate four questions before any records leave the company: whether you can export them, whether privacy law and your customer terms allow the intended use, who owns the content, and whether you hold the right to license it. De-identification does not answer the licensing question on its own.

For a support organization that goes ahead, SourceX runs the rights review, contracting, preparation and delivery of the case histories it licenses to AI developers. It works with companies in the U.S., Canada and the EU whose headcount peaked at 50 or more full-time employees, not counting contractors, and considers companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia only when at least 90% of the offered data is in English. That share is measured on the records actually offered, so a multilingual support archive needs the measurement before anything else. Before any of that, a short assessment, done without uploading files, shows whether the company qualifies.

Frequently asked questions

What should a translation fidelity and evidence retention test include?

It needs a frozen copy of the original-language record, a list of decision-critical facts extracted before reading the summary, a fact-by-fact score of the summary, a check of where each loss happened, a link to the case outcome and an inclusion decision. Two reviewers should be able to run it on the same case and agree.

Is a high machine translation score enough to trust a handoff summary?

No. Metrics such as BLEU, chrF and COMET score a translation against its source or a reference, not whether a much shorter summary kept the facts that drove the decision. Use them to screen translations, then check the summary by hand.

Who should review multilingual support cases?

A reviewer fluent in the source language extracts the facts, and a second reviewer scores the summary. A support lead who knows the product and policies should settle disagreements about which facts were decision-critical.

Should cases with lossy summaries be removed from a dataset?

Not always. When the original messages are kept and the loss is labeled, a lossy summary makes a useful failure example for evaluation. It should not be used as a gold summary for training.

How long should you wait before calling a case resolved?

Set a fixed outcome maturity window before sampling and apply it to every case. A case counts as resolved only when that window passes with no reopening and no repeat contact about the same issue.

Sources

  • SacreBLEU computes shareable BLEU, chrF, chrF++ and TER scores and prints a signature recording the settings used, so scores can be reproduced. Source
  • COMET provides reference-based and reference-free translation quality models scored from 0 to 1, and its XCOMET models flag error spans as minor, major or critical under the MQM typology. Source
  • The MQM error typology groups translation errors into high-level dimensions; its Accuracy dimension includes addition, omission and mistranslation. Source
  • Maynez et al. (ACL 2020) report from a large human evaluation that neural abstractive summarization models are highly prone to generate content unfaithful to the input document. Source
  • Zendesk's conversation translation in the Agent Workspace shows translated end-user messages with a Show original control that reveals the untranslated text. Source
  • Zendesk explains that the ticket event log keeps end-user comments in their original language and shows both the original and translated versions of agent messages. Source
  • Zendesk's Ticket Audits API is a read-only history of all updates to a ticket, where each audit can contain events such as comments and field changes with previous and new values. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies