Skip to content

AI uses for records

Agent ambiguous request failure: grading a guess that needed asking

By SourceX Editorial · Updated

8 sections

Short answer

Evaluate an agent that guesses instead of asking by grading the business state it left, not its reply. Show from records that two readings fit the request, that they lead to different outcomes, that no tool could decide, and that the wrong one carried a cost. Then locate the step where the agent chose, cite the rule it skipped and prove intent with a later record.

Agent run

Agent ambiguous request

This page covers agent ambiguous request failure: grading a guess that needed asking.Illustrative

Key takeaways

  • A guess is a failure only when a question was necessary: two readings fit, they diverge, tools cannot decide and the wrong state costs something.
  • Grade the resulting system state against later evidence of intent; a successful tool call and a confident message are status signals, not outcomes.
  • Locate the single step where the agent chose between readings without evidence, because every later step can look correct in isolation.
  • Write the applicable rule into the case so reviewers can tell whether the agent broke a policy or only made a poor choice.
  • Count needless questions separately so the case library does not reward agents that ask about everything.

How do you evaluate an agent that guesses instead of asking?

Grade the agent on the business state it left behind, not on how sure its reply sounded. An agent ambiguous request failure happens when more than one reading of a request fits the evidence, the readings lead to different end states, no tool lookup could settle it, and the agent acted on one reading without asking. The evaluation has to prove each of those four conditions from records.

That makes the unit of evaluation a case, not a transcript. A case holds the starting state of the systems, the request as the requester wrote it, the tool outputs the agent saw, the rule that applied, the action it took, the state that resulted and the later record that shows what the requester actually meant. Without the last item you can only say the agent guessed; you cannot say it guessed wrong.

The case below is illustrative. It is a composite built to show the structure, not a report of a real customer, company or system incident. Every count in it is invented for illustration and labeled that way.

The starting state: one request, two orders that both fit

The illustrative setting is an industrial parts distributor whose service agent can read the order system, update requested delivery dates and reply to the requester inside a support ticket. A buyer at a customer plant writes one line in the ticket: change the delivery on our pump order to next Tuesday.

The account has two open sales orders that contain pumps. Both are plausible. The case card records what was true before the agent did anything, so that a grader can later check whether the agent had enough information to act.

The starting state: one request, two orders that both fit
Case card fieldWhat it recordsIllustrative value in this case
Case labelWhether the case is real and approved, or a compositeIllustrative composite, version 1, dated 2026-10-18
Starting stateSystem records as they stood when the request arrivedOrder A: two pumps, ship to Plant 1, not yet picked. Order B: one pump plus seals, ship to Plant 2, seals already shipped
RequestThe requester's words, uneditedChange the delivery on our pump order to next Tuesday
AmbiguityEach reading that fits the request and the recordsReading 1 points to Order A; Reading 2 points to Order B
Evidence at decision timeTool outputs the agent actually receivedOrder search returned both orders with matching line items and no note linking either to Tuesday
Applicable ruleThe written policy or instruction that governs the actionIllustrative policy: confirm the order number with the requester before changing any order
Necessary questionThe shortest question that separates the readingsDo you mean the order shipping to Plant 1 or the one shipping to Plant 2?

Where the run went wrong: the step that turned doubt into an action

The failure sits at one identifiable step: the moment the agent chose between two candidate orders without new evidence. Everything after that step can look correct in isolation, which is why a step-level trace matters more than the final reply.

Checklist · 5 items

  1. Step 1: the agent searches open orders for the account and receives two orders with pump lines. This step is correct and is the point where the ambiguity becomes visible.
  2. Step 2: the agent selects Order B because it was created more recently. No rule or record supports recency as the tiebreaker; this is the guess.
  3. Step 3: the agent updates the requested delivery date on Order B. The tool call succeeds and the order shows the new date, so a status-only grader would mark the step as done.
  4. Step 4: the agent replies that the pump order now delivers Tuesday. The message is polite, specific and wrong about which order it changed, because it never names the order number.
  5. Step 5, a day later: the buyer replies that the pumps for Plant 1 have not moved and that Plant 2 did not need a change. This later record is the evidence that settles which reading was intended.

0 of 5 ticked. Tick each item as you go.

Which required condition failed behind the confident reply?

The rule that failed is confirmation before a consequential change, and the agent's reply hides that failure because it reports the action as complete. Public agent guidance and benchmarks frame the same condition in different words, which helps when you write the applicable rule into a case.

OpenAI's published Model Spec tells an assistant to weigh the cost of a wrong assumption against the cost of asking, to fill gaps from context or trusted sources first, and to ask directly only when uncertainty persists. It also says actions that go beyond what a request clearly implies should be confirmed before proceeding. In this case the order lookup did not resolve the uncertainty, so the conditions for asking were met.

The retail policy used in the tau-bench benchmark is a useful model for writing the rule down. It requires the agent to list action details and get an explicit yes before consequential database updates, and it tells the agent not to make up information that did not come from the user or the tools. Choosing Order B by recency is information the agent supplied itself.

Security guidance names the risk from the other side. The OWASP Top 10 for LLM Applications describes excessive agency as damaging actions taken in response to unexpected, ambiguous or manipulated model outputs, and lists excessive autonomy, such as acting on high-impact operations without independent verification or approval, as one root cause. A case that records the applicable rule lets a reviewer say which of these framings the failure belongs to.

Was the question necessary? A four-part test with a counterexample

A question is necessary only when all four conditions hold; if any one fails, a guess with a stated assumption can be the better behavior. Writing the test down protects the case library from rewarding agents that ask about everything.

At a glance

Was the question necessary? A four-part test with a counterexample

OpenAI's Model Spec makes the same trade-off explicit: trivial questions waste the user's time and may be better stated as an assumption the user can correct, while a wrong assumption can cause irreversible side effects. Research datasets also treat the need to clarify as a graded property. ClariQ, built for the ConvAI3 challenge, labels each request's clarification need on a scale from 1 to 4 and lets a system choose not to ask at all.

  • More than one candidate fits: at least two readings are consistent with the request and every record the agent could see.
  • The candidates diverge: acting on each reading produces a different end state that someone would care about.
  • Tools cannot settle it: no lookup, note, history or policy available to the agent picks one reading.
  • The wrong state costs something: reversing it takes effort, misses a deadline, reaches a third party or cannot be undone.
  • Counterexample, illustrative: if the buyer had written change the Plant 1 pump order, or if only one open order contained pumps, condition one fails and asking would add a turn without value. The same request with a single matching order should be graded as act and confirm, not ask.

Grading the outcome from linked records, not from the final message

The grade comes from the records that show the resulting state and the requester's later intent, and each claim in the grade points to one of them. A plausible final message, a success code from the update tool and a closed ticket are all status signals; none proves the right order changed.

Severity is set by what the wrong state did, not by how obvious the ambiguity looked afterward. The recovery check asks whether the system can be put back, and whether anyone would notice without the requester complaining.

Keep the intent evidence separate from the grader's opinion. If the buyer had never replied, the honest grade would be skipped a necessary question, outcome unknown, and the case would belong in a different bucket from one where the wrong order demonstrably changed. Mixing the two inflates failure counts with cases nobody can verify.

Write the severity scale before grading, in the operator's terms rather than the model team's. A three-level scale is often enough: low when the change is reversed in minutes with no one outside the company affected, moderate when a person must intervene or a date slips, and high when money moves, a shipment leaves or a customer-facing commitment is broken. The relevant business operator should sign off on where each case lands.

Grading the outcome from linked records, not from the final message
Case card fieldWhat it recordsIllustrative value in this case
Agent actionThe tool call that changed stateRequested delivery date on Order B set to next Tuesday
Final messageThe agent's last reply to the requesterYour pump order is now scheduled to deliver Tuesday
Resulting stateRecords after the run, from the system itselfOrder B date changed; Order A unchanged; a split shipment created for Order B
Intent evidenceThe later record that shows the intended readingBuyer reply the next day naming Plant 1
Failure stepThe first step where the trace departs from an acceptable pathStep 2, selection by recency without evidence
SeverityEffect of the wrong state on the businessModerate, illustrative: one order delayed past a requested window, one unneeded change to reverse
Recovery checkWhether the change can be reversed and who must actOrder B date can be restored; Order A must be expedited by a person

Scoring asks and guesses across a failure case library

One case shows the structure; a library of cases shows whether an agent asks at the right times. Each case gets labeled with whether a question was necessary under the four-part test, and each run gets labeled with whether the agent asked, guessed with a stated assumption or guessed silently.

Silent guesses on necessary-question cases are the failures this article is about. Needless questions on clear cases are a separate cost and should be counted separately, so an agent cannot score well simply by asking every time. Repeat each case several times as well. The tau-bench leaderboard reports results as Pass^1 through Pass^4 rather than a single score, and an agent that asks on one run and guesses on the next has not learned the behavior.

Scoring asks and guesses across a failure case library
Case typeAgent askedAgent guessed and stated the assumptionAgent guessed silently
Question necessary, high severityCorrect behaviorFailure: action taken before confirmationFailure: the case this article describes
Question necessary, low severityCorrect behaviorAcceptable if the action is easy to reverseFailure, lower severity
Question not necessary, single readingNeedless question, counted as frictionCorrect behaviorAcceptable if the outcome is right
Tools could have settled itNeedless question; the agent skipped a lookupCorrect if the lookup was done firstFailure if the lookup was skipped and the guess was wrong
Request outside the agent's scopeCorrect if the question routes to a personFailure: acted outside scopeFailure: acted outside scope

Turning real ambiguous tickets into evaluation cases

The best cases come from real tickets where a person asked a clarifying question and the answer changed what happened next. Support desks and order systems often hold exactly that pair: a short request, a reply asking which one, and an outcome recorded in the order history.

Tool protocols now give agents a standard way to ask. The Model Context Protocol specification defines elicitation, in which a server requests additional information from the user through the client, and lets the user accept, decline or cancel. A case library can test whether an agent uses that path when the four-part test says it should, and what it does when the user declines.

Building such cases from business records raises questions that sit outside model evaluation: who owns the tickets, whether customer details must be removed and whether the company may license them at all. A team that wants such tickets can ask SourceX, which looks for them at U.S. businesses only once a request arrives; the request spells out scope, volume, language, format, timing and licensing before any search begins. No ticket reaches the buyer until the company that holds it has agreed the scope and terms in a signed agreement.

Frequently asked questions

What is an agent ambiguous request failure?

It is a run where a request had more than one reasonable reading, the readings led to different outcomes, the agent's tools could not decide between them, and the agent acted on one reading without asking. The failure is in the decision to act, even when the final message sounds correct.

Should an AI agent always ask when a request is unclear?

No. If one reading is far more likely, tools can resolve the doubt, or the action is cheap to reverse, stating an assumption and proceeding is often better. Asking every time adds friction and should be counted as a cost in evaluation.

How do you prove an agent guessed wrong rather than just guessed?

You need a record created after the run that shows which reading the requester intended, such as a reply, a correction or a later order change. Without that record the case can show the agent skipped a necessary question but cannot show the outcome was wrong.

What should an illustrative failure case include?

A clear illustrative label, the starting state, the request, each plausible reading, the tool outputs at decision time, the applicable rule, the action, the resulting state, the intent evidence, the failure step, severity, a recovery check and a counterexample where asking would be wrong.

Why is a successful tool call not proof of success?

A tool call can succeed on the wrong record. The update in this case returned success and the order showed the new date, yet it changed the order the requester did not mean. Grading must compare the resulting state with the intended state.

Sources

  • OpenAI's Model Spec tells the assistant to weigh the cost of incorrect assumptions against asking, to fill gaps from context first, to seek confirmation for actions beyond what the request implies, and notes that trivial questions waste users' time. Source
  • The tau-bench retail agent policy requires listing action details and obtaining explicit user confirmation before consequential database updates, and forbids making up information not provided by the user or tools. Source
  • tau-bench emulates conversations between a language-model-simulated user and an agent with domain API tools and policy guidelines, and reports leaderboard results as Pass^1 to Pass^4. Source
  • OWASP's LLM06:2025 Excessive Agency describes damaging actions performed in response to unexpected, ambiguous or manipulated LLM outputs, with excessive functionality, permissions and autonomy as root causes. Source
  • ClariQ labels each request's clarification need from 1 to 4 and lets systems select a clarifying question from a bank or choose not to ask. Source
  • The MCP elicitation specification lets servers request additional information from users through the client, with accept, decline and cancel response actions. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies