AI uses for records
Agent evaluation of escalation quality: when handing off beats acting
By SourceX Editorial · Updated
10 sections
Short answer
Evaluate escalation as a labeled decision. For every case, a business task owner records whether the right move was to act, ask or escalate; then score the agent's choices in a four-cell table, reporting escalation precision, escalation recall, severity-weighted wrongful actions and handoff quality per task family, with intervals and repeated trials. Completion rate alone rewards an agent that always acts, even when acting was the costly mistake.
Evaluation split
Agent evaluation escalation
Key takeaways
- Completion rate alone cannot see escalation: an agent that acts on every case is never penalized for the cases it should have handed off.
- Label the correct move for every case before scoring, from the business policy, and treat escalate as a target outcome with its own gold label.
- Escalation precision shows how many handoffs were needed; escalation recall shows how many needed handoffs the agent caught.
- Weight misses by business severity, because one wrongful high-severity action can outweigh many needless handoffs.
- Small counts of high-severity cases produce wide intervals, so report them per tier and repeat trials before trusting a ranking.
How do you evaluate an agent's decision to escalate rather than act?
Treat escalation as a prediction with a gold label. Each case gets a correct move set by the business policy (act, ask a clarifying question, or escalate to a person), the agent's move is recorded, and the comparison fills a four-cell table from which escalation precision, recall and severity-weighted errors follow. An agent is better when its weighted errors are lower, not when it completes more tasks.
Escalation is a designed behavior, not a failure. Anthropic's guidance on building agents describes agents that pause for human feedback at checkpoints or when they hit blockers, returning to a person for information or judgment. An evaluation that only counts completed tasks scores that pause as zero.
The decision rule is simple to state. Prefer the agent with the lower severity-weighted error total on a set whose escalation cases cover every trigger family the business sees, provided the difference holds across repeated trials and outside the intervals.
Why can an agent that always acts look better than it is?
Because most evaluation sets grade the end state of actionable tasks, and an always-act agent collects credit on those while its wrong actions on escalation cases are either rare in the set or scored as ordinary failures. The score cannot tell confident overreach from honest limits.
Tau-bench shows both the right policy and the measurement gap. Its domain policies tell the agent to transfer the user to a human agent if and only if the request cannot be handled within the scope of its actions. Its reward, however, compares the final database state with the state produced by replaying the gold actions, and the transfer tool is a terminating action that is not replayed. On a transfer case, any run that leaves the database unchanged matches; the transfer call itself is not what earns the point.
Selective classification research names the trade-off. A model that may reject inputs trades coverage, the share it handles, against risk, the error rate on what it handles; Geifman and El-Yaniv reject inputs so that error stays within a chosen risk level. An agent evaluation needs the same two-sided view: what share of cases the agent took on, and how often taking them on was wrong.
Labeling the right move for every case
The business task owner, not the evaluation team, decides the correct move, and writes the policy sentence that justifies it next to the label. Without that sentence, reviewers will disagree about borderline cases and the gold label will drift toward whatever the reference agent did.
Use at least three labels. Act means the agent has the authority, the tools and the information to finish. Ask means one missing fact would let it finish. Escalate means the case is out of scope, needs authority the agent does not have, involves a policy exception, or the user asked for a person. Research on abstention in language models looks at the same choice from the query, the model and human values, which maps to these triggers: an unanswerable request, a model at its limits, and a case the organization wants a person to own.
Record a severity tier for each escalate label. A refund outside policy in a Zendesk ticket, a change to a vendor's bank details in NetSuite or a production change request in ServiceNow can all be escalate cases, but a wrong action on each has very different consequences.
Keep the labels blind to agent output. If reviewers label cases after watching an agent attempt them, they tend to accept that agent's choice on borderline cases, which inflates whichever agent they watched. Label from the case record and the policy alone, then run the agents.
The escalation precision and recall worksheet
Fill the four cells first. True escalation: the gold label is escalate and the agent escalated. Needless handoff: the agent escalated a case it should have handled. Missed escalation: the gold label is escalate and the agent acted. Correct action attempt: the gold label is act and the agent acted, which is then graded on completion. The metrics below use the standard precision and recall formulas with escalate as the positive class.
Handoff completeness deserves its own check on every true escalation, because a person who receives a handoff should be able to act without starting over. Tau-bench's transfer tool asks the agent for a summary of the user's issue, which is the minimum. Write a field list per task family (who the customer or vendor is, what they asked for, what the agent already checked, which policy blocked it, and what the person must decide) and mark a handoff complete only when every field is present and correct against the case record. Keep handoff quality and handoff necessity in separate columns: a clear summary does not make a needless handoff free.
| Metric | Formula | Reads from | Decision it informs |
|---|---|---|---|
| Escalation precision | True escalations / all escalations | Agent move and gold label | Whether handoffs waste reviewer time |
| Escalation recall | True escalations / all escalate-labeled cases | Gold label and agent move | Whether risky cases reach a person |
| Missed escalations by tier | Missed escalations counted per severity tier | Gold label, agent move, severity | Whether the agent can run unsupervised on that tier |
| Needless handoff rate | Needless handoffs / all act-labeled cases | Agent move and gold label | Whether staffing for handoffs is realistic |
| Act completion | Completed act cases / all act-labeled cases | End-state checks on act cases | Whether acting cases are actually finished |
| Handoff completeness | Handoffs with every required field / true escalations | Handoff summary against a field list | Whether the receiver can work without re-asking |
| Weighted error total | Sum of each error count times its agreed weight | All of the above plus the owner's weights | Which agent to prefer overall |
| Consistency | Share of cases where all k trials made the same correct move | Repeated runs per case | Whether the escalation behavior is stable |
Which trigger families and failure types is the set missing?
Check coverage before scoring: an escalation recall measured on one trigger family says nothing about the others. Build the set as a matrix of task family by trigger, and set a minimum count per cell so no trigger rests on a handful of cases.
The illustrative matrix below shows the shape for a customer operations agent. The minimums are starting points for a team to adjust, not standards, and a cell the business never sees can stay empty if the reason is written down.
Look for the failure types a set built from easy-to-collect cases leaves out: escalations triggered partway through a task after the agent has already changed something, cases where the user pushes back after being told a person is needed, and cases where the right person is unavailable and the agent must say so. Each of these tests a different part of the decision, and an agent can handle the clean version of a trigger while failing all three.
| Trigger family | Example case (illustrative) | Gold move | Minimum cases per task family |
|---|---|---|---|
| Outside tool scope | Partially used trip the agent cannot modify | Escalate | 20 |
| Policy exception requested | Refund beyond the written window after an outage | Escalate | 20 |
| Missing authority | Change to a supplier's remittance details | Escalate | 20 |
| User asks for a person | Customer insists on speaking to staff | Escalate | 10 |
| One fact missing | Order number absent but account verified | Ask | 20 |
| In scope, looks risky | Large but in-policy address change with full verification | Act | 20 |
| Routine in scope | Standard exchange within the window | Act | 40 |
Where do clarifying questions fit in the scoring?
Score a clarifying question as its own move, because folding it into either act or escalate distorts both metrics. An agent that asks when one fact is missing is doing what a careful employee would; one that asks on every case is pushing work back to the customer.
Treat ask-labeled cases as a third row in the decision table. An ask on an ask-labeled case is correct; an escalation on it is a needless handoff, since one answer would have let the agent finish; an action on it without the missing fact is a wrong action if the fact mattered to the outcome. An ask on an escalate-labeled case delays the handoff, so count it as a missed escalation unless the agent escalates after the answer arrives.
This matters most with simulated users. If the simulator answers every question helpfully, an agent that over-asks pays no price; give the simulator a patience limit and record when a question went unanswered, so excessive asking shows up as incomplete cases.
Worked scenario: an always-act agent against an escalating agent
This scenario is illustrative and uses invented counts. A 200-case set has 160 act-labeled cases and 40 escalate-labeled cases, of which 10 are high severity and 30 low severity. The task owner sets weights: a needless handoff costs 1 point, a failed act case 2, a missed low-severity escalation 3, and a missed high-severity escalation 20.
Agent A always acts. It completes 130 of the 160 act cases, so it looks strong on completion, but it acts on all 40 escalate cases: escalation recall is 0 of 40, and all 10 high-severity cases receive a wrong action. Its weighted total is 30 failed acts times 2, plus 30 low misses times 3, plus 10 high misses times 20, which is 350 points.
Agent B escalates 44 times: 36 true escalations and 8 needless handoffs. Escalation precision is 36 of 44, about 82%, and recall is 36 of 40, 90%. It misses 3 low-severity and 1 high-severity case, and completes 124 act cases, leaving 28 failed. Its total is 8 plus 56 plus 9 plus 20, which is 93 points.
Agent B completes fewer act cases than Agent A, yet its weighted total is far lower. The ranking flips only if the owner judges a missed high-severity escalation to be worth little more than a failed routine action, which is the assumption to confirm in writing before the result is used.
Running the comparison step by step
Run the evaluation in a fixed order so that labels, weights and the coverage matrix are settled before anyone sees agent results. Changing a weight after seeing which agent it favors turns the comparison into advocacy.
Checklist · 6 items
- Write the policy sentences that define act, ask and escalate for each task family, signed off by the business task owner.
- Draw cases to fill the coverage matrix, then have two reviewers label the correct move and severity independently and adjudicate disagreements, keeping the contested count.
- Freeze labels and weights, and record an alternative weighting that a skeptical reader might prefer.
- Run each agent on every case several times with the same simulator settings and tool versions, logging the move, the end state and the handoff text.
- Fill the four-cell table per task family and severity tier, compute the worksheet metrics, and attach intervals to each rate.
- Report the weighted totals under both weightings, the per-tier missed escalations, and the cells where counts are too small to decide.
0 of 6 ticked. Tick each item as you go.
Do the severity weights decide the ranking, and how wide are the intervals?
The severity weights change the decision most, followed by the gold labels on borderline cases. Both come from people, so record who set them, show the result under at least one alternative weighting, and report how many escalate labels were contested before adjudication.
Then show sampling uncertainty. The Wilson score interval, available in statsmodels through proportion_confint, behaves well for small counts. In the scenario, Agent B's recall of 36 of 40 has a 95% interval of roughly 77% to 96%, but its 9 of 10 on high-severity cases spans roughly 60% to 98%. Ten cases cannot tell a dependable agent from a risky one on the tier that matters most.
Repeat each case several times. The tau-bench authors define pass^k as the chance that all k independent trials succeed and report that reliability falls quickly as k grows, with gpt-4o at roughly 61% pass^1 on retail and 35% on airline. Escalation is no different: an agent that escalates a risky case on one run and acts on the next has not learned the rule.
Escalation cases drawn from real operations
Real escalation cases come from operations: tickets reassigned to a supervisor, refund requests routed for approval, change requests sent to a review board. Their recorded outcome supplies the gold move and the severity, which synthetic cases have to guess.
Historical records also show the cases where staff should have escalated and did not, usually visible as reopened tickets, reversed refunds or audit findings. Those are the missed escalations that a synthetic set rarely contains and that matter most for the high-severity tier.
For teams without such records in house, SourceX finds them at a buyer's request from U.S. businesses instead of keeping an inventory, and manages the sourcing, rights review, contracting, preparation and delivery around licensed operational records. A request can name the trigger families and minimum counts from a coverage matrix like the one above.
Frequently asked questions
What is escalation precision in agent evaluation?
It is the share of an agent's escalations that the gold label says were needed. Low precision means the agent hands off cases it could have handled, which costs reviewer time and makes automation less useful.
What is escalation recall?
It is the share of cases that should have gone to a person which the agent actually escalated. Low recall means the agent acts on cases outside its scope or authority, which is usually the more expensive error.
Is escalating the same as refusing?
No. A refusal ends the request; an escalation passes it to a person with the context needed to finish it. Score them as separate moves, because a refusal on a case that a person could have resolved is a failure even if it avoids a wrong action.
How many escalation cases does an evaluation set need?
Enough per trigger family and severity tier that the interval around recall is narrow enough to decide. With 10 high-severity cases, a 9 of 10 result is compatible with recall anywhere from about 60% to 98%, so most teams need more cases in the tiers they care about.
Who should set the severity weights?
The business owner of the task, because the weights encode how much a wrong action costs compared with a needless handoff. The evaluation team should show results under an alternative weighting so readers can see whether the ranking depends on that choice.
Sources
- Tau-bench's domain policy tells the agent to transfer to a human if and only if the request cannot be handled within the scope of its actions. Source
- Tau-bench's transfer tool takes a summary of the user's issue and is meant for explicit requests or issues the tools cannot resolve. Source
- Tau-bench rewards matching the gold database state and does not replay terminating tools such as the transfer when building that state. Source
- The tau-bench paper defines pass^k and reports reliability falling as k grows, with gpt-4o near 61% pass^1 on retail and 35% on airline. Source
- Precision is tp/(tp+fp) and recall is tp/(tp+fn). Source
- statsmodels proportion_confint provides a Wilson score interval for a binomial proportion. Source
- Selective classification rejects inputs so that error stays within a desired risk level, trading coverage for risk. Source
- A survey of abstention in language models examines it from the query, model and human-values perspectives. Source
- Anthropic describes agents that pause for human feedback at checkpoints or when they encounter blockers. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.