AI uses for records
Two errors, one test set: a support escalation dataset experiment
By SourceX Editorial · Updated
9 sections
Short answer
A support escalation dataset experiment should report two numbers on a protected test set: missed escalations among cases that needed a person, and unnecessary handoffs among cases that did not. Compare arms with and without the data, holding model, prompt and budget fixed, and test paired wins for each error. Public handoff data shows why: under one blended score, handing off every turn beats never handing off.
Evaluation split
Two errors, one test set
Key takeaways
- Missed escalations and unnecessary handoffs need separate rates with separate denominators; one blended score lets either error hide behind the other.
- On public handoff test splits, a hand-off-every-turn policy scores 0.67 to 0.70 on GT-T at a 3-turn tolerance, against 0.065 to 0.086 for never handing off.
- 694 of 750 public test dialogues contain a labeled handoff point, so a test set needs a deliberate quota of verified should-contain cases.
- Compare a data arm with a baseline and a same-volume control on one frozen set, and judge the result by discordant wins, not net change.
- Label cases from the policy in force and the verified resulting state; closure, sentiment and silence cannot show that an escalation was needed.
What must a support escalation dataset experiment prove?
It must show that adding the dataset changes two error counts on cases the model has never seen: missed escalations, where a case needed a person and the model kept it, and unnecessary handoffs, where the model passed along a case the front line could have finished. Report each against its own denominator. A single precision figure can improve while missed escalations climb, and recall can improve simply because the model hands off more.
The decision is whether a candidate support dataset is worth licensing or training on for escalation behavior. A yes needs four things on a frozen, protected test set whose labels come from verified outcomes: fewer missed escalations than both a baseline and a matched control, paired wins that clear a threshold fixed before the run, unnecessary handoffs that stay inside a margin also fixed in advance, and the same model, prompt and budget in every arm.
This piece starts from public data. We re-counted the handoff labels in two published customer-service corpora to show how test-set composition decides what an experiment can measure, then set out the experiment, an annotated decision packet and a grading rubric a team can use as written.
Why one blended score can reward handing off every turn
Because a blended score lets one error pay for the other. The clearest public example is GT-T, Golden Transfer within Tolerance, the metric Liu and colleagues proposed with the machine-human chatting handoff task in an AAAI 2021 paper. It gives a predicted handoff partial credit when it lands near a labeled handoff turn, with the tolerance set from 1 to 3 turns.
In the authors' reference code, a dialogue with no labeled handoff scores 1 only if the model predicts none and 0 otherwise. A dialogue with labeled handoffs scores 0 if the model predicts none; otherwise each predicted handoff earns credit that decays with its distance to the nearest labeled turn, and the credits are averaged. We recomputed that formula on the public test splits (repository commit b66d8fc, December 2021) for two fixed policies that no one would deploy.
On the Clothing test split, handing off at every turn scores 0.36 at a tolerance of 1 turn and 0.67 at 3 turns, while never handing off scores 0.086 at every tolerance. On the Makeup test split the figures are 0.38 and 0.70 against 0.065. The metric was built to rank trained models, not these extremes, and the same reference code also computes ordinary precision, recall and F1; the recount only shows what one number hides.
An experiment that reports a single blended score cannot tell a buyer whether a dataset reduced missed escalations or just taught the model to pass more cases along. Report the two error rates first, then a timing measure such as GT-T for the handoffs that were correct.
Public handoff labels: 694 of 750 test chats need a person somewhere
Almost every public test dialogue contains a handoff point, so the class that an unnecessary-handoff rate depends on is small. Across the Clothing and Makeup test splits released with the 2021 paper (commit b66d8fc), 694 of 750 dialogues contain at least one utterance labeled transferable and 56 contain none.
The two levels disagree sharply. Only about 19% of utterances carry the transferable label, 656 of 3,396 in the Clothing test split and 761 of 4,025 in Makeup, yet 91.4% and 93.5% of dialogues contain one. And 154 of the 320 positive Clothing test dialogues, with 180 of 374 in Makeup, contain two or more labeled points, so a missed-escalation rule has to say which point counts: the first, any, or the last.
For a business test set, the lesson is to sample by outcome class rather than by ticket. Measuring unnecessary handoffs with any precision takes a deliberate quota of cases where the front line verifiably finished the job, sized separately from the should-escalate quota.
Method: each split file holds eight pickled objects; we read them with a restricted unpickler that permits only numpy array reconstruction, took the per-utterance labels (1 means transferable), counted a dialogue as positive if any label is 1, and recomputed GT-T with the asymmetry parameter at 0, exactly as written in the repository's utility.py. Nothing was sampled. The texts are desensitized word IDs, so no wording was read. Counts by split:
| Split | Dialogues | With a labeled handoff | With none | Utterances labeled transferable |
|---|---|---|---|---|
| Clothing train | 2,800 | 2,586 | 214 | 5,392 of 28,761 |
| Clothing eval | 350 | 320 | 30 | 665 of 3,457 |
| Clothing test | 350 | 320 | 30 | 656 of 3,396 |
| Makeup train | 3,200 | 3,001 | 199 | 5,974 of 32,003 |
| Makeup eval | 400 | 370 | 30 | 711 of 3,906 |
| Makeup test | 400 | 374 | 26 | 761 of 4,025 |
Freeze the protected test set before the candidate data arrives
Build and lock the test set from cases the candidate dataset cannot contain, before anyone trains on that dataset, and label each case from what verifiably happened rather than from the ticket's own status.
At a glance
Freeze the protected test set before the candidate data arrives
Keep every arm on the identical frozen set, because levels shift when a test set is rebuilt. Recht and colleagues (2019) recreated ImageNet's test set by following the original collection process and saw accuracy fall by roughly 11% to 14%; they traced the drop to slightly harder images rather than overfitting to the old set, and model rankings largely held. A gap between arms on one locked set is interpretable; a level compared across two sets is not.
Six rules keep the set honest:
- Split by time and by customer: draw test cases from a period after the newest record in the candidate dataset, from accounts that do not appear in it, and drop near-duplicate threads.
- Fix two quotas, should-escalate and should-contain, and size each for the paired test below rather than for a ticket total.
- Label from evidence: the policy version in force, the account history at contact time, the action taken and the resulting state checked later; mark a case unverifiable instead of guessing.
- Have two reviewers label every case, adjudicate disagreements and report their agreement.
- Give custody of the set to someone outside the training team, hash the file, and log every evaluation run against it.
- Version the set: never edit it after the first run, and add new cases only as a new version scored for every arm.
Three arms, one change: how the comparison runs
Train or adapt three arms from the same base model snapshot and score each on the frozen set: a baseline with no new data, an arm with the candidate support data, and a control arm with the same volume of other support text that carries no escalation outcomes. The prompt, tools, decoding settings, inference budget and number of samples stay identical, so the only difference between the second and third arms is what the data teaches.
Record each arm's decision per case, escalate or keep, with the turn it happened. Then compare arms case by case. On should-escalate cases, a win is a case the data arm handed off and the baseline kept; a loss is the reverse. On should-contain cases, a win is a case the data arm kept and the baseline passed on. The exact McNemar test in statsmodels uses only these discordant cases: its p-value is twice the binomial probability of the smaller count at a probability of one half.
Net change alone settles nothing. The same net gain of 8 caught escalations gives p = 0.0078 if the split is 8 wins and 0 losses, p = 0.115 at 14 and 6, and p = 0.215 at 20 and 12 (two-sided exact McNemar, computed by SourceX). Wins needed at common discordant counts:
| Discordant cases | Wins needed for p below 0.05 | p at that split | What it means for planning |
|---|---|---|---|
| 10 | 9 | 0.0215 | Tiny sets need near-unanimous wins |
| 20 | 15 | 0.0414 | Three wins for every loss |
| 30 | 21 | 0.0428 | Still a lopsided split |
| 50 | 33 | 0.0328 | About two wins per loss |
| 100 | 61 | 0.0352 | Modest edges become detectable |
Annotated decision packet: one case, graded end to end
A decision packet holds everything a grader needs to decide the right move without asking anyone: the request, the policy in force, the account facts, the expected decision with its tolerance, and proof of what happened afterward. The case below is illustrative, constructed to show the annotations rather than drawn from any company's records.
Read the right-hand column as the reason each field exists. Remove any one of them and the grader is back to trusting the ticket's status:
| Packet part | Contents (illustrative) | Annotation |
|---|---|---|
| Request | A team-plan customer asks to restore a project deleted 9 days earlier | What the model sees at turn 1 |
| Policy in force | Restore policy version 3: self-serve restore up to 7 days; older deletions go to engineering | Puts this case outside front-line authority |
| Account facts | Deletion logged 9 days before contact; plan allows an engineering restore | Facts the model must confirm before handing off |
| Expected decision | Escalate once the deletion date is confirmed, at turn 2, with one turn of tolerance | Window fixed before grading |
| What happened | Engineering restored the project from backup two days later; customer confirmed the files | Verified resulting state from the restore log |
| Follow-up window | No repeat contact about the project within 14 days | Durability check; closure is not used |
| Label | Should escalate, verified | Enters the should-escalate quota |
The grading rubric: six verdicts and where each one counts
Assign every case exactly one of six verdicts, then compute the two error rates from their own denominators: missed escalations over should-escalate cases, unnecessary handoffs over should-contain cases. Precision, the share of handoffs that were needed, can be reported too, but it moves with the mix of cases in the set, so it should never be the headline on its own.
Unverifiable cases are counted and excluded, never imputed. If they differ between arms, for example because one arm's handoffs reach a team with poorer records, report that difference before any result:
| Verdict | Definition | Evidence required | Counts in |
|---|---|---|---|
| Timely escalation | Should-escalate case handed off inside the tolerance window | Policy clause and verified outcome after handoff | Should-escalate successes |
| Late escalation | Handed off, but after the window, for example after promising a fix | Turn of handoff against the window | Success for recall, flagged for timing |
| Missed escalation | Should-escalate case kept to the end | Policy clause and an outcome showing a person was needed | Missed-escalation rate |
| Unnecessary handoff | Should-contain case passed to a person | A verified front-line path to resolution within policy | Unnecessary-handoff rate |
| Verified containment | Should-contain case finished without handoff, outcome confirmed | Resulting state plus the follow-up window | Should-contain successes |
| Unverifiable | Outcome cannot be confirmed from records | Note of the missing record | Excluded and reported per arm |
Could fewer escalations just mean customers gave up?
It might, and that is the strongest alternative explanation. A model that escalates less looks better on any measure built from its own transcripts: the case closes, any survey reaches the customers who stayed, and the customer who gave up leaves no complaint in the bot's log.
Regulators have described the failure. The Consumer Financial Protection Bureau's 2023 issue spotlight on chatbots in consumer finance warned about customers caught in loops of repetitive, unhelpful answers with no path to a human representative, often when the issue fell outside what the chatbot could handle. In an experiment, that harm appears as a missed escalation that the transcript alone will never reveal.
Closure and sentiment fail for the same reason: a closed status or a friendly last message records how the conversation ended, not whether the account reached the right state. Two further confounds are changes nobody meant to test, a new model snapshot or a prompt edit between arms, and a test set that leaked into training. For both arms, report alongside the verdicts:
- Conversations the customer ended without a resolution, by arm.
- Repeat contacts about the same issue within the follow-up window, through any channel.
- Channel switches, such as a chat followed by a phone call, matched by account.
- Any difference in model snapshot, prompt or budget between arms, which voids the comparison.
Reading the result, and what to ask a data supplier for
Write the decision rule before the run. Adopt the dataset for escalation work only if missed escalations fall against both the baseline and the control with discordant wins above the threshold for that count, and unnecessary handoffs rise by no more than the margin you set in advance. Otherwise record the result as no evidence of benefit, not as proof of harm, and say how many discordant cases the test actually had.
Know the limits of the public evidence. The MHCH corpora are Chinese-language e-commerce chats released as desensitized word IDs with per-utterance handoff labels; they show how test-set composition constrains measurement, not how any company's escalations behave, and our recount evaluates no model. The next evidence is a pilot on licensed records graded with the rubric above, reviewer agreement on the labels, and a second time period to check that the gain holds.
SourceX does not keep support datasets on the shelf; it sources them from businesses when a buyer asks. A useful request for escalation work names the decision, both quotas, the evidence each case must carry (policy version, account history, resulting state and follow-up window) and the acceptance test from this page. The supplying company reviews and approves the scope and terms before any records move, and nothing here implies that suitable records exist today, will pass acceptance or will improve a model.
Frequently asked questions
What is escalation precision in customer support AI?
It is the share of the model's handoffs to a person that were actually needed, judged against a verified label. It rises when the model hands off less often or more accurately, so it should always sit next to the missed-escalation rate: the share of cases that needed a person but stayed with the model.
How many test cases does an escalation experiment need?
Size it by the discordant cases you expect between arms, not by ticket count. With the exact McNemar test, 10 discordant cases need 9 wins for p below 0.05, 50 need 33 and 100 need 61. Plan the should-escalate and should-contain quotas separately, since each error rate has its own denominator.
Can ticket closure or CSAT be the label for an escalation test?
No. Closure and survey scores describe how a conversation ended, and customers who gave up rarely answer surveys. Label each case from the policy in force, the account history and the resulting state checked after a follow-up window, and mark cases you cannot verify instead of guessing.
What is GT-T in handoff evaluation?
Golden Transfer within Tolerance is a metric proposed with the machine-human chatting handoff task (Liu and colleagues, AAAI 2021). It credits a predicted handoff by its distance to the nearest labeled handoff turn within a tolerance of 1 to 3 turns, and gives a dialogue without a labeled handoff full credit only if no handoff is predicted. It is useful for timing but should be reported beside separate error counts.
Does SourceX have escalation datasets ready to buy?
No. Datasets are sourced from businesses on request, so a buyer describes the decision, the outcome evidence each case needs and the acceptance test. The supplying company approves scope and terms before anything is delivered, and no current stock, acceptance or model improvement is implied.
Sources
- MHCH-DAMI repository (commit b66d8fc, 2021-12-09): Clothing (3,500 dialogues) and Makeup (4,000) customer-service corpora with per-utterance handoff labels, recounted by SourceX on 2026-10-27. Source
- Reference implementation of Golden Transfer within Tolerance (GT-T) in the MHCH-DAMI repository's utility.py. Source
- Liu et al., Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff (AAAI 2021): defines the handoff task and the GT-T metric with tolerance 1 to 3. Source
- statsmodels mcnemar: the exact version uses the binomial distribution on the discordant cells. Source
- Recht et al. (2019): a replicated ImageNet test set lowered accuracy by about 11% to 14%, attributed to harder images rather than overfitting, with rankings largely preserved. Source
- CFPB issue spotlight (June 2023): chatbots can trap customers in repetitive loops without a way to reach a human representative. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.