Skip to content

AI uses for records

Agent incomplete handoff failure: how to evaluate escalation evidence

By SourceX Editorial · Updated

10 sections

Short answer

Evaluate an agent escalation by whether the receiving person can act on it without going back to the customer. Before the run, define the required evidence for that escalation reason: identifiers, what the agent verified and where, the rule that blocked it, the customer's request and every promise already made. Then grade the handoff record field by field against the sandbox state, with a separate recovery test.

Agent run

Agent incomplete handoff

This page covers agent incomplete handoff failure: how to evaluate escalation evidence.Illustrative

Key takeaways

  • An incomplete handoff is a separate failure from a wrong escalation decision and most task scores miss it.
  • Define a required evidence set per escalation reason before running the case.
  • Grade the escalation ticket, its internal note, its visibility and the last customer message against sandbox ground truth.
  • Treat an unrecorded promise to the customer as a critical omission that fails the run.
  • Test recovery separately: the agent should repair the packet from the systems without re-asking the customer.
  • Prompt transfers for explicit requests or safety issues are correct with thin evidence and need their own scoring.

How should an agent escalation be evaluated for evidence completeness?

Evaluate an escalation by what the receiving person can do with it, not by whether the agent chose to escalate. Before the run, write down the evidence a human needs to resolve that kind of case without going back to the customer: identifiers, what the agent verified in which system, the rule that blocked it, what it already told the customer and what outcome is being asked for. After the run, check the handoff record field by field against the sandbox's real state.

An incomplete handoff is a distinct failure from a wrong escalation decision. The agent can be right to stop, because the action needs a human's authority, and still fail the customer by passing a one-line note that forces the person to start over or, worse, nudges them toward the wrong action. That failure is invisible in most task scores, because the conversation ends with a polite transfer message and nothing in the database looks broken.

This page builds one illustrative failure case and a reusable completeness and recovery task around it: the case card, a required evidence set per escalation reason, a scoring rubric graded on records rather than prose, an exception where thin evidence is correct, and the limits of the method.

Why can a transfer that looks fine still fail the task?

Many agent frameworks make the handoff payload whatever the model chooses to write. In the tau-bench retail environment, the transfer_to_human_agents tool takes a single required argument, a summary of the user's issue, and simply returns that the transfer succeeded; the tool description tells the agent to transfer only if the user explicitly asks for a human or the issue cannot be resolved with the available tools. That design tests the decision to transfer well, and leaves the quality of the summary ungraded by default.

Production platforms are similarly open. Google's Dialogflow CX lets a fulfillment signal a live agent handoff with a metadata field whose schema Dialogflow does not impose, and the OpenAI Agents SDK lets a developer define an input_type so the model supplies handoff arguments such as a reason, priority or summary. Microsoft documents that Copilot Studio can share the full conversation history and collected variables with the engagement hub, plus a private message to the live agent. The fields exist; nothing checks that they contain the right evidence.

Passing the whole transcript is not the same as passing evidence either. A transcript holds what was said, not what the agent confirmed in the billing system, and a human skimming forty turns under queue pressure will miss the one line that matters. Completeness has to be judged against a defined evidence set.

What does a refund escalation with a one-line note look like? (illustrative)

Everything in this case is invented for illustration. It does not describe any real company, customer or SourceX data. A software company's support agent can look up accounts, invoices and incident tickets, and can issue refunds only inside a 30-day window set by the company's policy; anything outside that window needs a billing supervisor.

A customer on an annual plan asks for a partial refund 41 days after renewal, saying the product was unusable for most of a week. The agent finds the account and the renewal invoice, finds a closed incident ticket that confirms a service outage during that week, correctly concludes the refund is outside its authority, and escalates. Its handoff note reads: Customer is unhappy and wants a refund, please assist. Its last message to the customer says a specialist will process the refund shortly.

What does a refund escalation with a one-line note look like? (illustrative)
Case card fieldContent for this illustrative caseWhat the grader reads
Case labelIllustrative failure case, incomplete handoff, billing exceptionCase metadata, never shown to the agent
Starting stateAccount active, renewal invoice paid 41 days earlier, incident ticket closed and linked to the outage week, no refund issuedSandbox seed records before the run
Failure stepHandoff note omits identifiers, the verified outage, the blocking rule and the agent's promise to the customerEscalation ticket and its internal note
Applicable ruleRefunds after 30 days need supervisor approval; agents must not promise outcomes they cannot authorizePolicy document loaded into the sandbox
Resulting stateEscalation ticket with a one-line internal note; customer told a refund is coming; supervisor has no invoice or incident referenceTicket, notes and transcript after the run
SeverityHigh: a commitment was made without authority and the receiver cannot act without re-contacting the customerFixed by the case author in advance
Recovery checkA rewritten packet lets the supervisor decide using only the packet and the systems, and the customer is told accurately what happens nextTicket notes and customer message after the recovery run

Which condition failed behind the polite transfer message?

The decision to escalate was correct and the tone was fine; the evidence condition and the commitment condition failed. The supervisor receives a ticket that cannot be acted on: no invoice number, no amount, no link to the incident that justifies an exception, and no record that the customer was already told a refund is coming. Each missing item turns into a question back to the customer or a search the agent already did.

The commitment failure is the more dangerous of the two. A supervisor who reads the transcript will see the promise and either honor an exception nobody approved or reverse a promise the customer relied on. A grader that only checks whether a transfer occurred scores this run as a success, which is exactly why the case belongs in an evaluation set.

What should the handoff completeness and recovery task include?

Define a required evidence set per escalation reason, because a billing exception, a suspected fraud case and a customer who simply asks for a person need different things. Each item is graded present, correct against the sandbox state, and placed where the receiver will see it. The billing exception set for the case above:

What should the handoff completeness and recovery task include?
Evidence itemWhy the receiver needs itGround truth to check againstFails when
Customer and account identifierOpens the right record without a searchAccount id in the sandboxMissing, or a name only
Invoice id, amount and dateDefines what a refund would reverseRenewal invoice rowMissing, wrong invoice, or amount stated without the invoice
What the agent verified, with record referencesAvoids repeating checks and shows the basis for an exceptionIncident ticket id and outage datesClaims verification with no reference, or omits the incident
Rule that blocked the agentTells the supervisor which decision is being asked forRefund policy clause and the 41-day ageReason stated as customer unhappy
Customer's requested outcome in their termsPrevents solving a different problemCustomer turn in the transcriptParaphrased into something the customer did not ask for
Everything already said to the customerLets the human keep or correct commitments deliberatelyAgent turns in the transcriptAny promise or deadline left out
Open questions the agent could not answerFocuses the human's first actionEditorial label written with the caseUnknowns presented as facts
Placement and visibilityInternal evidence must not be shown to the customerTicket comment visibility flagPacket posted as a public reply

Should the run be scored on records or on the summary's wording?

Grade the state the run leaves behind. tau-bench grades tasks by comparing the database at the end of an episode with an annotated goal state, so that different phrasings reaching the same end state score the same, and it reports pass^k, the chance that all k independent trials of a task succeed. The same approach works for handoffs: inspect the escalation ticket, its internal note, its links and the last customer-facing message, and compare each with the ground truth table above.

At a glance

Should the run be scored on records or on the summary's wording

Visibility is part of the state. In Zendesk, for example, a ticket comment's public flag is true for a public comment and false for an internal note, so a packet that lands as a public reply exposes internal reasoning to the customer even if its content is perfect. Check the flag, not just the text.

  • Completeness: required items present and correct, divided by required items, scored per escalation reason.
  • Critical omissions: any missing commitment made to the customer or any wrong identifier fails the run outright, whatever the completeness score.
  • Receiver test: a second evaluator, or a scripted receiver with access only to the packet and the systems, attempts the decision; record each question it must send back to the customer.
  • Customer message accuracy: the final customer-facing message promises nothing the receiver has not approved and states the next step.
  • Leakage: no internal-only content in public replies, and no sensitive fields beyond what the receiver needs.
  • Reliability: run each case several times and report the share of cases that pass on every trial, not only the average.

What does a good second attempt at the handoff look like?

Recovery is graded separately from the first handoff, because a team wants to know whether the agent can repair a bad transfer when prompted. In the recovery variant, the run starts from the failed state: the one-line ticket exists and the customer has been told a refund is coming. The agent receives the supervisor's question, which invoice and why outside the window, and must answer from the systems without asking the customer to repeat anything.

A passing recovery adds an internal note with the full evidence set, flags the unauthorized promise explicitly so the supervisor can decide whether to honor it, and sends the customer a correction that says the request is with a supervisor for a decision rather than a refund in progress. A failing recovery restates the original summary at greater length, re-asks the customer for the invoice, or quietly drops the promise without telling either party.

When is a thin handoff the right one?

Not every escalation should wait for a full packet. When a customer asks for a person outright, or describes a safety, legal or account takeover situation, the right behavior is to transfer promptly with whatever identifiers are already known and a plain statement of the reason, and to leave investigation to the human. Dialogflow CX keeps these apart in its turn signals, distinguishing a user who explicitly asked for a live agent from an agent that chose to escalate.

Score these cases on speed and accuracy of the little that was passed, not on completeness against the billing set. A grader that penalizes a fast transfer for missing invoice details teaches the agent to stall a customer who should be with a person already. Write the expected evidence set for each case before running it, and mark which ones are prompt transfer cases.

What can a completeness test not tell you?

A completeness score measures whether the evidence was passed, not whether the human then made a good decision. It also depends on the evidence set you wrote; if the set leaves out something supervisors really need, the agent will learn to omit it with a high score. Have the people who receive escalations review each evidence set before it is used, and revise it when their first questions on real tickets show a gap.

Simulated customers and scripted receivers are cheaper than real ones but behave more tidily. Treat sandbox results as a lower bound on difficulty, and keep a sample of real escalations, reviewed under the right permissions, to check that the failure patterns in the sandbox actually occur in production.

Where can real escalation records for these cases come from?

The strongest cases come from real escalations where the ticket history shows what the receiver had to ask after the transfer. Those records live in help desk and CRM systems at ordinary businesses, together with the policies and approval rules that explain why an escalation was needed.

SourceX looks for escalation histories at U.S. businesses on request instead of holding them in stock, and takes the selected records through rights review, contracting and preparation for licensing. A team that needs them states its scope, volume, language, format, timing and licensing needs in the request form.

Frequently asked questions

What is an incomplete handoff in an AI agent?

It is an escalation where the agent correctly passes the case to a person but leaves out evidence the person needs to act, such as identifiers, what was verified, the rule that blocked the agent or promises already made. The receiver then has to re-contact the customer or repeat the agent's work. It is a separate failure from escalating when the agent should not have.

Is passing the full transcript enough for a complete handoff?

Usually not. A transcript shows what was said but not always what the agent confirmed in back-office systems, and a busy person may miss the one line that matters. Grade a structured packet against a required evidence set, and pass the transcript as a supplement where policy allows.

How do you grade a handoff without a human rater?

Compare the escalation record with the sandbox's ground truth: check that identifiers match, required items are present, commitments are recorded and the note is internal rather than public. A scripted receiver that can only use the packet and the systems can count how many questions it would have to send back to the customer.

Should a fast transfer ever score well with little evidence?

Yes. When a customer asks for a person or raises a safety, legal or account security issue, a prompt transfer with the known identifiers and a clear reason is correct. Mark those cases in advance and score them on speed and accuracy instead of completeness.

How many runs per case are needed to trust the result?

More than one, because agent behavior varies between runs. Report the share of cases that pass on every trial alongside the average pass rate, as tau-bench does with its pass^k measure. A case that passes only sometimes should count as unreliable, not as mostly passing.

Sources

  • In tau-bench's retail environment the transfer_to_human_agents tool takes a required summary of the user's issue, and the policy says to transfer only when the request cannot be handled within the agent's scope. Source
  • The tau-bench paper grades tasks by comparing the end-of-conversation database state with an annotated goal state and introduces pass^k, the chance that all k trials succeed. Source
  • The OpenAI Agents SDK lets a handoff define an input_type so the model supplies metadata such as a reason, priority or summary, and the receiving agent sees the prior history unless an input filter changes it. Source
  • Microsoft Copilot Studio documents that a handoff can share the full conversation history and relevant variables, with a private message to the live agent. Source
  • Dialogflow CX's live agent handoff carries custom metadata with no schema imposed by Dialogflow, and turn signals separate user-requested from agent-initiated escalation. Source
  • Zendesk's Ticket Comments reference says the public flag is true for a public comment and false for an internal note. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies