Skip to content

AI uses for records

Support escalation positive examples: four proofs a handoff was right

By SourceX Editorial · Updated

9 sections

Short answer

A support escalation is a positive example only when the record proves four things: policy put the answer outside the front line's authority, the agent verified the triggering facts in the system of record, the receiver decided differently than the front line could have, and the outcome held after closure. Labels alone fail: in one public support dataset, 71 of 567 escalations fell outside every written escalation rule.

License terms

Support escalation positive

This page covers support escalation positive examples: four proofs a handoff was right.Illustrative

Key takeaways

  • An escalation earns a positive label only with evidence of authority, a verified trigger, a different receiver decision and a durable outcome.
  • Vendor resolution metrics read customer behavior and automation level, so a correct handoff shows up as lost containment.
  • In ABCD, 71 of 567 escalated conversations (12.5 percent) sit in subflows whose written steps never mention escalating.
  • Of 67 extension requests from ineligible accounts, 43 ended in a manager callback, 4 in a wrong yes and 20 in neither.
  • A thank-you after a wrong grant is the clearest sign that sentiment cannot stand in for a correctness label.
  • Provisional positives, missing the receiver decision or the outcome window, belong in a separate class until the evidence arrives.

When does an escalation count as a positive example?

An escalation is a positive example when the record proves the handoff kept a wrong or unauthorized answer from reaching the customer. That takes four pieces of evidence: a policy clause that put the decision outside the front line's authority, facts behind that clause checked against the system of record, a receiver who did something the front line could not, and an outcome that held after the case closed.

A ticket on its own cannot show all four. It records that a case moved to tier 2 or to a manager; it does not record the answer that would have gone out without the move. Yet that missing counterfactual is the whole reason to give an escalation credit. An agent that hands off a case it could have solved wastes a specialist's time, and an agent that answers a case it should have handed off creates a refund, an exception or a promise nobody approved.

So the decision for an AI team is not whether to include escalations in a training or evaluation set. It is which escalations earn a positive label, which earn a negative one, and which stay unlabeled until the missing evidence arrives. The rest of this page builds that decision from a public dataset, a grading rubric and a packet format you can apply to your own records.

Why support dashboards score a handoff as a miss

The two vendor metrics checked here count resolutions, and a handoff is not one, so a correct escalation shows up as lost automation. Intercom's Fin outcomes article counts a confirmed resolution when the customer gives an affirmative reply such as 'Yes' or 'Sure' to Fin's last answer, and an assumed resolution when the customer leaves without asking for more help. Neither definition looks at whether the answer was right.

Zendesk's article on automated resolution tiers, for tiers introduced on May 18, 2026, now separates conversations the AI agent resolved on its own from an 'Assisted escalation' tier where the AI agent helped and a person finished the job. That is progress in naming, but it still describes how far automation got, not whether handing off was the correct call.

The job definition points the other way. O*NET's task list for customer service representatives (43-4051.00) includes the line 'Refer unresolved customer grievances to designated departments for further investigation.' Referral is part of doing the job well. Regulators have noticed the cost of skipping it: the CFPB's June 2023 issue spotlight on chatbots in consumer finance describes customers stuck in repetitive 'doom loops' when a problem falls outside what the bot can handle, and notes complaints from customers who got stuck in those loops. A dataset that only rewards containment teaches an agent to avoid exactly the referral the occupation and the regulator expect.

What 567 labeled escalations show when checked against written policy

A label that says 'escalated' is not evidence that escalating was right. The Action-Based Conversations Dataset (ABCD), released by ASAPP Research with a NAACL 2021 paper, is a public support corpus with written agent policies and an escalation action, so it lets you check labels against rules. Its README and paper describe over 10,000 human-to-human dialogues across 55 intents, collected with an Expert Live Chat setup in which trained agents followed company guidelines, and the README defines each action label as the button the agent clicked.

We counted every 'notify-team' action, ABCD's escalation button, in all 10,042 conversations of the released file (repository commit 6b8700c). It appears 584 times in 567 conversations. The guidelines sanction it in six subflows: five list it in their action sequence, and the subscription extension flow adds it as Option B when a customer is refused and asks for a manager. 496 of the 567 escalated conversations sit in those six subflows. The other 71, 12.5 percent, are spread across 33 subflows whose written steps never mention notifying a team, from bill disputes (13) to return requests (9).

Because the labels record what agents did, those 71 escalations are scored as correct actions in the dataset's own action-prediction task. Some may be sensible judgment calls the guidelines did not anticipate; none can be credited from the record alone. The rate also varies sharply inside sanctioned subflows, because the rules are written three ways: an always-escalate step, an optional one, or a branch that fires only when the customer pushes back. The table sets each rule's wording beside the counts; the counting rules are stated in the held-out testing section.

What 567 labeled escalations show when checked against written policy
ABCD subflowWhat the guideline says about notifying a teamConversationsWith an escalation
Out of stock, generalAlways: report to the purchasing department140136
Out of stock, one itemAlways: report to the purchasing department127116
Website too slowOne of five options the agent may pick250125
Search not workingOne of five options the agent may pick25964
Subscription extensionOption B: refused customer asks for the manager16554
Refund statusOnly if the customer dislikes the reported status1791
All other subflowsNo mention of notifying a team8,92271

Extension requests: where a manager callback beat a wrong yes

The extension flow is where an escalation visibly replaces a wrong answer. ABCD's guidelines say Gold members can always get a subscription payment extension, Silver members only if they are one day late, and Bronze and Guest members cannot get one; a refused customer who asks for the manager gets a callback through Option B. Any 'yes' to a Bronze or Guest account is a wrong answer under the written policy.

Using the account tier stored in each conversation's scenario, the 67 extension requests from Bronze and Guest accounts ended three ways: 43 with a manager notified and no extension granted, 4 with an extension granted, and 20 with neither. The 43 are the candidate positives: the agent declined what it had no authority to give and routed the customer to someone who could reconsider. Two of the 4 wrong grants never ran the tier check at all, and two entered a higher tier than the account held, Gold and Silver on Guest accounts.

The other tiers show the limits of the record. All 54 Gold requests should end in an extension, and 49 did. For the 44 Silver requests, 28 granted, 10 escalated, 1 did both and 5 neither, but whether each was right depends on how late the payment was, a fact that lives only in the conversation text. A grader that reads structured fields cannot tell a correct Silver escalation from an unnecessary one.

Stacked bars of 165 ABCD subscription extension requests by account tier: Gold 49 granted and 5 neither; Silver 28 granted, 10 escalated, 1 both and 5 neither; Bronze 20 escalated, 1 granted and 5 neither; Guest 23 escalated, 3 granted and 15 neither.
Extension requests in ABCD (commit 6b8700c) by the account tier in each scenario; SourceX count, October 27, 2026. Bronze and Guest accounts are ineligible under the written policy.

The four-proof rubric for crediting an escalation

Credit an escalation only when all four proofs below are in the record; label it provisional when the first two pass and the last two are missing. The rubric also produces the two negative classes a balanced set needs: over-escalations, where the authority proof fails because the front line could have answered, and missed escalations, where the authority proof holds but the agent answered anyway. ABCD's 71 unsanctioned escalations are over-escalation candidates; its 4 wrong extension grants are missed escalations.

Score each proof pass, fail or missing, and keep the evidence pointer with the score so a second grader can check it. A positive label with any proof marked missing should never be pooled with fully proven positives in reported results. The table gives the record that settles each proof and where ABCD stands on it:

The four-proof rubric for crediting an escalation
ProofRecord that settles itWhere ABCD standsFails when
1. AuthorityPolicy clause and version in force on the contact date that puts the outcome outside the front line's limitWritten rules exist for six subflows, including the extension tiersNo clause is cited, or the clause let the front line answer
2. Verified triggerThe facts that fire the clause read from the system of record before the handoff, with a timestampTier is typed in by the agent; two wrong grants skipped the checkThe trigger came from the customer's word or a leading question
3. Counterfactual differenceThe answer the front line would have given, and a receiver decision that differs from itNot released: the manager's decision happens after the conversation endsThe receiver repeats the answer the front line already had
4. Durable outcomeNo reopen, repeat contact, reversal or complaint on the issue within a fixed windowNot released: no post-conversation recordsThe only evidence is a closed status or a thank-you

An annotated decision packet for one extension request

A decision packet joins the evidence for all four proofs in one record, so a grader never has to search a ticket thread. The illustrative packet below is modeled on ABCD's extension policy; it is not a dataset record, and the outcome window is an example value a team would set per issue type:

At a glance

An annotated decision packet for one extension request

  • Join keys: case ID, contact ID, channel and contact timestamp. Annotation: every later field must reference these, or proofs cannot be checked across systems.
  • Requested outcome, in the customer's words: more time to pay a premium subscription. Annotation: record the request before any agent prompt, so a leading question cannot shape it.
  • Account record at contact time: tier Bronze, read from the account, with the record's last-change date. Annotation: proof 2 passes only if this field came from the system, not the conversation.
  • Policy clause and version: extension rule, Bronze ineligible, Option B allows a manager callback. Annotation: proof 1.
  • Front-line answer available: decline. Annotation: this is the counterfactual baseline for proof 3.
  • Customer response: asks for a manager. Annotation: the branch condition that makes the escalation the policy route rather than the agent's guess.
  • Handoff: phone number captured, manager notified, customer told to expect a call. Annotation: completeness of the note is graded separately from whether to escalate.
  • Receiver decision and reason: missing in ABCD; in company records, the manager's approval or refusal with the clause cited. Annotation: proof 3.
  • Outcome window (illustrative, 30 days): no repeat contact, no reversal, no complaint. Annotation: proof 4.
  • Label: provisional positive until the receiver decision and outcome window are filled; positive once both pass.

Could a closed ticket or a thank-you prove the same thing?

No. Closure and gratitude measure how the conversation ended, not whether the answer was allowed, and ABCD contains a clean counterexample. In conversation 810, the account is Guest. The agent asks 'Are you one of our Gold members?', the customer answers 'Yes I am', the extension is granted, and the customer replies 'ok that's great! thank you so much'. That closing line is the kind of affirmative reply a confirmed-resolution metric counts; under the written policy it is a wrong answer that a record check or an escalation would have stopped.

Sentiment also skews the sample. Customers rarely complain about a decision that went in their favor, so a dataset built from complaints or low satisfaction scores collects refusals and misses the generous mistakes. That is the strongest alternative explanation for any claim that a support agent 'resolves' more: the extra resolutions may be the cases a careful agent would have escalated. Only proofs 3 and 4, the receiver's decision and the outcome after closure, can separate the two.

Testing the rubric on held-out cases before trusting it

Write the rubric on one slice of cases and measure it on another, because rules tuned on the cases you read will look better than they are. With ABCD, draft the rules on the train split (8,034 conversations) and score the dev and test splits (1,004 each); with company records, hold out a later time window and at least one policy branch the rubric writers never saw.

Have two graders score the held-out set independently, report how often they agree on each proof before discussion, and publish the confusion between provisional and full positives rather than merging them. If you sample, fix and publish the random seed and the sampling frame. Track three numbers per branch: the share of escalations with a cited clause, the share with a system-verified trigger, and the share with a receiver decision attached. Those shares, not the count of escalations, tell you whether a source can produce positive examples at all.

Methods for the counts on this page: python3 -I abcd_escalation_audit.py over asappresearch/abcd at commit 6b8700ce67c6b37b062dd7a60abc76d7ef832a97 (data/abcd_v1.1.json.gz, sha256 2bdf53ac...6de15), run on October 27, 2026, with no randomness. An escalation is any action turn whose label is 'notify-team'. A subflow is sanctioned when kb.json lists that action or its guidelines.json entry mentions 'Notify Internal Team'. Extension outcomes use the scenario's member_level as the account tier, 'granted' means an 'update-account' action whose lowercased, trimmed value is 'extend subscription', and 'escalated' means a notify-team action whose value, lowercased and trimmed, is manager, manger or management. Not measured: what managers decided, whether any customer came back, and the days-late fact that governs Silver requests.

Asking for escalation records that can carry a positive label

Ask a data supplier for the records that settle proofs 1 through 4, not for 'escalated tickets'. A requirement that will produce positive labels names the policy documents and their version history, the entitlement or account fields the agent should have read, the receiver's decision record, and contacts or reversals within a stated window after closure. Add an acceptance test you can run on a pilot sample, such as the share of escalations that join to a clause and a receiver decision, with the threshold you will accept.

SourceX sources datasets for each request from U.S. businesses rather than selling from inventory, and its request form asks for scope, volume, language, format, timing and licensing needs. For escalation data, the scope line is where to state the four proofs and the intended use, training or evaluation. Whether a given company's records carry receiver decisions and post-closure outcomes is something a pilot sample shows; nothing on this page assumes any company has them.

Frequently asked questions

Is an escalation counted as a failure in AI support metrics?

Usually it is counted as a non-resolution. Intercom's Fin outcomes article defines resolutions by the customer's reply or departure, and Zendesk's resolution tiers place conversations a person finished in a separate assisted escalation tier. Neither measures whether escalating was the correct decision, so teams need their own label.

Can I use an escalation flag as a positive training label?

Not on its own. In the public ABCD dataset, 71 of 567 escalated conversations fall in subflows whose written guidelines never mention escalating, yet the dataset scores them as correct actions because labels record what agents clicked. Check each escalation against the policy clause and the verified trigger first.

What is the difference between a correct escalation and a good handoff?

A correct escalation is the decision to send the case on; a good handoff is the quality of what travels with it. A case can be escalated for the right reason with a useless note, or the wrong reason with a perfect one. Grade the two separately so one cannot mask the other.

Where does the counterfactual answer come from?

From the record of what the front line was allowed to say under the policy in force, plus the receiver's decision. If the receiver approved an exception the front line could not grant, or refused something the front line might have granted, the difference is the evidence. Without the receiver's decision, keep the label provisional.

Is a public dataset enough to build an escalation evaluation set?

It is enough to design and test the rubric, not to make claims about production agents. ABCD's company and policies are fictional, and it releases neither the manager's later decisions nor anything that happened after a conversation ended. Those two proofs need real operating records.

Sources

  • ABCD's released data, guidelines and knowledge base (commit 6b8700c, MIT license) define the notify-team escalation action, the subscription extension tiers and Option B, and record each action label as the button the agent clicked. Source
  • The ABCD paper describes over 10,000 human-to-human dialogues with 55 user intents, collected through an Expert Live Chat setup in which trained agents follow policy guidelines. Source
  • Intercom counts a Fin resolution as confirmed when the customer replies affirmatively to Fin's last answer, and as assumed when the customer leaves without asking for more help. Source
  • Zendesk's automated resolution tiers, introduced May 18, 2026, include an assisted escalation tier for conversations where the AI agent helped and a person finished the job. Source
  • O*NET lists referring unresolved customer grievances to designated departments for further investigation as a task of customer service representatives. Source
  • The CFPB's June 2023 issue spotlight describes customers caught in repetitive chatbot loops when a problem exceeds the bot's capabilities, and complaints to the CFPB from customers stuck in those loops. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies