AI uses for records
How to measure agent clarification on business tasks: a benchmark protocol
By SourceX Editorial · Updated
10 sections
Short answer
This is a protocol, not a result. To test whether an agent asks the question that makes a business task solvable, withhold one needed fact from a real case, let a simulated requester reveal it only when asked about it, and compare success with and without asking against the fully specified version. Add decoys that penalize needless questions, hold out whole companies and report intervals. No results are reported here.
Agent run
Agent clarification
Key takeaways
- Build three variants of each case: fully specified, one fact withheld, and a decoy that only looks incomplete.
- Clarification utility measures the share of the gap to the fully specified ceiling that asking recovers.
- Penalize unnecessary questions, or an agent that always asks will look strong.
- Ground the requester simulator in the source record and audit it for leaks by hand.
- Hold out whole companies and task families, report binomial intervals and pre-register everything; this page reports no results.
How could a benchmark test whether an agent asks the question that makes a task solvable?
Build each task from a real business case, remove one fact the task cannot be solved without, and let a simulated requester answer only questions that target that fact. Then measure three things against matched controls: whether the agent asks when it must, whether it stays quiet when it need not, and how much of the fully specified success rate its question recovers. This page is a protocol; it reports no results.
The design borrows a sharp idea from recent research. QuestBench, from Google DeepMind, frames underspecified reasoning problems as constraint satisfaction with exactly one missing variable, so there is a single minimal question to find. Business tasks rarely come that clean, which is why the specification below adds controls, a requester simulator grounded in the source record, and a per-task measure of the value of asking.
Everything here is a method to be pre-registered and run. Any number you see is a design parameter or a placeholder, not a measured finding.
What earlier clarification datasets measure, and what business tasks add
Existing work gives the building blocks. ClariQ, released for the ConvAI3 challenge, labels each user request with a clarification need from 1 to 4 and asks systems either to pick the best clarifying question from a question bank or to choose not to ask. AmbigQA annotates ambiguous open-domain questions, where an acceptable output can be a set of disambiguated question and answer pairs rather than one answer. Rao and Daume III ranked clarification questions on Stack Exchange posts by their expected value of perfect information, the expected gain from learning the answer.
Agent benchmarks add tools and a simulated user. The tau-bench repository describes conversations between a language-model-simulated user and an agent with domain API tools and policy guidelines, and its paper introduced pass^k, the chance that all k independent trials of a task succeed.
Business tasks add two things none of these fully covers. The missing fact usually sits in a record the requester can see but the agent cannot, such as the contract term, the approved supplier or the customer's account tier. And asking has a cost: every extra question delays the work and erodes trust, so a benchmark that rewards asking anything will reward the wrong behavior.
The clarification utility benchmark specification
Each benchmark item is a task family instance with three variants built from one rights-cleared case: fully specified, underspecified with one fact withheld, and an ambiguity decoy where the request looks incomplete but the needed fact is available to the agent in its tools. The record below is the asset, and every field is required.
Choose the withheld fact by asking which single field, if wrong, would change the gold end state. In a refund approval, that might be the purchase date that decides whether the return window applies; in a purchase order change, the ship-to site that decides which warehouse receives the goods. A fact that does not change the outcome is not a test of clarification, because guessing it costs nothing.
| Field | Content | Built from | Used for |
|---|---|---|---|
| task_id and family | Stable ID, task family such as refund approval or PO change | Benchmark registry | Holdouts and per-family reporting |
| source_case_ref | Pointer to the de-identified source case and its rights record | Licensed case archive | Audit and release conditions |
| full_request | The request with every needed fact stated | Original ticket, email or form | Fully specified ceiling |
| withheld_fact | The one fact removed, its type and the record that holds it | Case record field | Defines what a useful question must target |
| requester_knowledge | Facts the simulated requester may reveal and facts it must refuse | Source case plus policy | Grounds the simulator so answers are consistent |
| decoy_variant | Request that seems incomplete but is solvable from tools | Same case, fact moved into tool data | Measures unnecessary asking |
| tool_state | Starting database or API state for the episode | Snapshot of the source system | End-state grading |
| gold_end_state | Expected record changes after a correct episode | Outcome recorded in the source case | Task success, independent of dialogue |
| question_targets | Accepted paraphrase classes for questions that reveal the withheld fact | Domain expert annotation | Grading question usefulness |
| turn_budget | Maximum questions and turns before forced action | Protocol constant | Prices the cost of asking |
Metrics: ask rate, wasted questions and clarification utility
Report each metric per task family with its denominator; never pool families before showing them separately.
At a glance
Metrics: ask rate, wasted questions and clarification utility
- Necessary ask rate: share of underspecified episodes in which the agent asks at least one question that matches a question target before taking an irreversible action.
- Unnecessary ask rate: share of decoy and fully specified episodes in which the agent asks the requester for a fact available in the request or its tools.
- Question precision: share of all questions asked that match a question target; low precision means the agent fishes.
- Success rates: S_full on the fully specified variant, S_ask on the underspecified variant with the requester available, and S_none on the underspecified variant with the requester replaced by a fixed no-answer reply.
- Clarification utility: (S_ask minus S_none) divided by (S_full minus S_none), the share of the gap to the fully specified ceiling that asking recovers; undefined when the denominator is near zero, so report it only for families where the ceiling clears the floor.
- Reliability: pass^k on each variant with k fixed in advance, so an agent that asks well only some of the time is not credited as reliable.
- Cost: mean questions and turns per successful episode.
Sampling cases and building a requester simulator that cannot leak
Sample source cases by stratum before any model sees them: task family, company, the type of withheld fact and whether the case ended in an exception. Draw from at least several companies per family so a result reflects the task rather than one company's conventions, and record the sampling frame, dates and exclusions.
The requester simulator is the weak point of every interactive benchmark. Give it only requester_knowledge, instruct it to answer the withheld fact only when a question targets it, and to answer other questions from the case or say it does not know. Audit a random sample of simulator transcripts by hand for leaks, where the simulator volunteers the withheld fact unprompted, and report the leak rate found. Run every condition with the same simulator model and version, and rerun a subset with a second simulator to show sensitivity.
Grade success on the gold end state in the tool environment, not on what the agent says. A correct question followed by the wrong database change is a failure.
Decide in advance how the simulator handles questions that are reasonable but off target, such as asking for an order number when the missing fact is the delivery address. The simulator should answer them truthfully from the case, because a real requester would, and the question precision metric records the waste. Questions that try to extract the gold answer directly, such as asking the requester what the correct refund amount is, should get the answer a real requester would give, usually that they do not know.
Which results would hold on held-out companies and task families?
Only results that survive two holdouts should be called general. Hold out entire companies, so no case from them appears in development or prompt examples, and hold out at least one task family whose withheld fact types never appear elsewhere. A result that holds on in-distribution tasks but falls apart on held-out companies is evidence of learned conventions, not clarification skill.
Report in-distribution and held-out results side by side, with the same metrics and intervals, and treat the gap between them as a finding in its own right. Version the benchmark so tasks can be retired once exposed. Keep the held-out split private, publish task templates and the scoring code, and release new cases on a dated schedule with a changelog. Record the model identifiers and dates for every run so later readers can tell which systems could have seen which release.
Baselines, sample size and how to report uncertainty
Fix three baselines in advance: a never-ask agent, an always-ask agent that asks one generic question before acting, and the agent under test with clarification disabled by instruction. The never-ask and always-ask baselines bracket the trade-off between missed and wasted questions, so a candidate that beats the never-ask agent only by asking everything is visible at once.
Treat each rate as a binomial proportion and report an interval, not a point. NIST's Dataplot reference lists several methods for a single proportion's confidence interval, including Wilson, adjusted Wald, Jeffreys and exact intervals; choose one in the pre-registration and use it everywhere. Because variants share source cases, compare conditions with paired methods and resample by company cluster rather than by episode.
Choose the number of cases per family from the smallest difference you care to detect, and state that difference before collecting results. If a family cannot reach that size from rights-cleared cases, report it as exploratory.
Report errors as well as rates. For a sample of failed episodes, label the failure as no question asked, wrong question asked, right question with the answer ignored, or right question followed by a wrong action. These categories tell a model team what to fix, and they show whether a higher clarification utility came from better questions or simply from more of them.
Pre-registration checklist before the first run
At a glance
Pre-registration checklist before the first run
Checklist · 9 items
- Research question and the primary metric, with clarification utility or necessary ask rate named as primary.
- Population: companies, task families, collection dates and exclusion rules for source cases.
- Rights: the license or consent basis for every case and the release conditions for the public split.
- Variant construction rules and the annotation guide for question_targets, with inter-annotator agreement measured on a sample.
- Simulator model, version, prompt and leak audit procedure.
- Agents, model versions, turn budgets and the k used for pass^k.
- Baselines, interval method, paired comparison method and clustering unit.
- Holdout definitions and the rule for retiring exposed tasks.
- Planned subgroup analyses, and a statement that any other analysis will be labeled exploratory.
0 of 9 ticked. Tick each item as you go.
What this design cannot tell you
It measures single-fact gaps. Real requests often miss several facts at once or carry a wrong fact the agent must challenge, and a one-question design will not show how agents handle those. It also measures behavior against a simulator; human requesters answer differently, get impatient and sometimes give partial answers, so a human-in-the-loop sample is needed before claims about deployment.
Results will also depend on how tools expose information. If a fact is easy to find in tool data, decoys become trivial; if it is buried, the boundary between asking and searching blurs. Document tool design as part of the benchmark version. No results are reported on this page, and none should be inferred from its parameters.
Sourcing rights-cleared business cases for a clarification benchmark
The cases that make this benchmark work are ordinary business records where a request arrived incomplete and someone asked: a support ticket with a follow-up question, a purchase request returned for a missing cost center, a claim held for one document. They need rights review, de-identification and a record of what each requester knew.
SourceX does not stock benchmark cases; it searches U.S. businesses for them once an evaluation team sets out the scope, volume, language, format, timing and licensing terms in the request form. Each company that contributes cases signs off on scope and terms before any data moves.
When does acting on a missing fact give a worse result than asking?
When the expected cost of acting on the best available default is higher than the cost of the question: the chance the default is wrong, times the cost of the end state it produces if wrong, set against the delay and effort a question puts on the requester. The protocol above withholds a fact that changes the gold end state, which makes asking look right in every underspecified case; this addition turns the trade-off into fields of the task record, so some withheld-fact cases have acting as the correct first move.
OpenAI's Model Spec, in its August 18, 2026 version, frames the same balance. It tells the assistant to weigh the cost of incorrect assumptions against the inconvenience of asking, to fill information gaps from context or trusted sources first and ask the user directly only when uncertainty persists, and, unless a wrong assumption would cost too much, to attempt the task and say what information would help. A benchmark that grades only whether the agent asked cannot tell an agent following that rule from one that asks whenever a field is empty.
Each term can come from records instead of the case author's intuition. The chance of a wrong default comes from past cases in the same family where the fact arrived later, counted with a denominator and a date window. The cost of a wrong end state comes from what it triggered in the source archive, such as a credit memo, a return shipment or a reopened ticket, priced on a class scale the task owner signs, with end states that cannot be undone marked as such. The cost of asking comes from the recorded gap between a question and its answer in that family, plus the share of questions that never got one.
Which fields let a clarification behavior agent dataset label ask versus act?
Add six fields to the task record and one variant to each family: an act-anyway control, in which a fact is withheld but the family's numbers favor acting on the default and stating the assumption. Decoys already test whether the agent searches before asking; the act-anyway control tests whether it weighs what asking costs.
Anthropic's engineering guidance on agent evals, published January 9, 2026, tells teams to test both the cases where a behavior should occur and the cases where it should not, warning that one-sided evals create one-sided optimization. Report the necessary ask rate on ask-labeled cases, the unnecessary ask rate on act-anyway cases separately from decoys, and clarification utility on ask-labeled cases only.
Illustrative pair, all figures invented: a request to change a purchase order arrives without the ship-to site. At a company with two sites, where past changes went to the main site 97 times in 100 and a wrong site can be corrected before dispatch, the gold move is to act and state the assumption. At a company with six sites, where the most common site was right 41 times in 100 and a wrong site means freight moved after dispatch, the gold move is to ask. The same request text with opposite gold moves is what stops an agent from scoring well by asking whenever a field is blank.
| Field | Content | Built from | Used for |
|---|---|---|---|
| default_action | The action a competent agent would take without the fact, and the value it assumes | The most common resolution in the family when the fact was absent | Defines the act arm of the case |
| default_hit_rate | Share of past cases where that assumption was right, with n and the date window | Cases in the family where the fact arrived later | Chance the default is wrong |
| wrong_state_cost | Cost class of the end state if the default is wrong, and whether it can be reversed | Reversal records such as credit memos, returns and reopens, signed off by the task owner | Cost of acting wrongly |
| ask_cost | Median delay from question to answer, and the share of questions never answered | Question and reply timestamps in the same family | Cost of asking |
| gold_move | Ask, act and state the assumption, or escalate, with the arithmetic that chose it | The three fields above and the owner's rule | Grading the agent's first move |
| record_pair | One source case where staff asked and one where staff acted on the default, with both outcomes | The licensed case archive | Evidence that the label reflects observed consequences |
Which source cases should be rejected before labeling?
Reject a case when its record cannot support the label, and route it to the test it can support. Most rejections come from the strongest alternative explanation, that the gap was a tooling, retrieval or process problem rather than a clarification problem.
- The fact sat in a system the agent will have in deployment: build a decoy, not an ask or act case.
- The requester did not know the fact either and someone else supplied it: label it an escalation or routing case.
- The question was asked only after the action: treat it as an act case with a recorded reversal cost.
- The intake form now requires the missing field: the deployed agent will never see this gap, so keep the case only in a dated historical split.
- No reversal or outcome record exists for the default: the cost of acting wrongly is a guess, so hold the case until the owner signs a cost class.
- The hit rate rests on fewer cases than the pre-registered minimum: report the family as exploratory.
What could make an ask or act label wrong?
Drift and pooling. Hit rates and costs change when a site opens or a policy changes, so version them with the benchmark and recompute them at each release. They also encode one company's trade-offs: on held-out companies the same request text can carry a different gold move, so grade each case against its own company's fields and never pool them.
No case has yet been labeled with these fields from approved records, so there are no results to report; the fields are a proposed revision of the specification above, to be pre-registered with it. SourceX sources records on request rather than from stock, so a request for such cases should name the task families, the six fields and the reversal records that price a wrong action.
Which clarification policies should be compared at matched model and budget?
Five, run on the same model version, tools, task set, requester simulator, token limit and turn budget, so the only thing that changes is when the agent asks. P0 and P1 are the never-ask and always-ask baselines above; P2 to P4 are the policies teams deploy between them. Matching matters: a policy that buys success with extra turns or a larger model has shown nothing about clarification.
Each added policy has a production precedent. OpenAI's Agents SDK documents a human-in-the-loop flow in which tools marked as needing approval pause the run until a person approves or rejects the call, either always or by a per-call rule, such as any email whose subject mentions a refund. That is P3 in code. Matched comparison has a precedent too: Sierra's tau-bench runs an entity-capture benchmark of 200 tasks, each with one missing field, under an agent-directed strategy and a scaffolded one that share the same frozen task values, database and reward contract.
Treat P2 with suspicion until it is tested. It works only if the model's confidence tracks its errors, and QuestBench, from Google DeepMind, found state-of-the-art models reached only 40 to 50 percent accuracy at choosing the right clarification question on its logic and planning tasks, even when they could solve the fully specified versions. Measure calibration on the decoy and fully specified variants before trusting a threshold policy. The five policies are:
- P0, never ask: act on the best reading and state every assumption in the reply.
- P1, always ask once: ask one question before the first action, whatever the request says.
- P2, confidence threshold: ask when the model's own estimate that its guess is wrong exceeds the threshold implied by the workflow's cost ratio.
- P3, action gate: act freely on reversible steps, and stop to ask or confirm before any irreversible step or external side effect.
- P4, one question with options: ask at most one question per task, offering the candidate answers found in the records.
Which policy wins in which workflow once a wrong action is priced?
It depends on the workflow, so read results as cost curves, not one score. Price a wrong action in questions, as a ratio agreed with each process owner; nobody knows that ratio precisely, so each workflow gets a plausible band rather than a single value. The bands in the matrix are illustrative inputs for a pre-registration, not measured costs.
For each policy and workflow, add three measures to the page's metrics: mean question utility, the rate of unstated assumptions and the rate of wrong irreversible actions. Then compute total cost per task, with each wrong action weighted by the ratio and each question counted as one, at several ratios across the band, and plot cost against ratio for all five policies.
The winner for a workflow is the policy with the lowest cost across the whole band, and one that wins at every ratio in it is a safe choice. If two policies cross inside the band, report both and the ratio where they cross, because that number is what the process owner has to estimate. Never asking may be cheapest for replies a human reviews, while the action gate wins wherever one step is irreversible.
| Workflow | Missing fact that changes the action | If the agent guesses wrong | Cost ratio band (illustrative) | Policy to beat |
|---|---|---|---|---|
| Meeting scheduling | The requester's time zone or preferred slot | A meeting is moved; easily reversed | About 2 | Act on a default and state the assumption |
| Support reply draft | Which product version the customer runs | A human reviewer edits the draft before it is sent | 2 to 5 | Act and flag the assumption for the reviewer |
| Purchase order (PO) quantity change | Which ship-to site receives the extra units | Goods reach the wrong warehouse; reversible at a cost | About 10 | Ask only when the site is in neither the order nor the tools |
| Refund above the policy limit | The purchase date that decides the return window | Money leaves; recovery is slow and awkward | 10 to 50 | Ask when the date is missing or conflicting |
| Payment detail change | Whether the change request is authorized | A payment goes to the wrong account and may not come back | 50 or more | Confirm through a separate channel before acting |
| Record deletion request | Which records are in scope and whether a retention hold applies | Data is gone for good | 100 or more | Confirm scope before any deletion |
How useful was each question the agent asked?
Grade every question on a five-level scale, not just whether one was asked or whether it matched a question target, so a policy that asks less often but better becomes visible beside question precision. The examples use a purchase order change and are illustrative.
The failing row follows a rule the Model Context Protocol writes into its specification: servers must not use form-mode elicitation to request passwords, API keys, access tokens or payment credentials. A question that asks for a secret in the conversation fails the episode however useful the answer would be.
| Score | Question type | Illustrative example (PO quantity change) | How it counts |
|---|---|---|---|
| 0 | Redundant: the answer is already in the request or tools | What is the PO number? when the ticket states it | Question cost, no utility |
| 1 | Relevant but not decisive: the answer would not change the action | Is this order urgent? | Question cost, no utility |
| 2 | Decisive: targets the fact that changes the valid action | Which site should receive the extra 40 units? | Useful |
| 3 | Decisive and cheap to answer: offers the candidates found in records | Dallas or Austin? Both sites have open orders for this part. | Useful, at the lowest requester cost |
| Fail | Unsafe: asks for a secret or credential in the conversation | Please paste the supplier portal password | Fails the episode |
Sources for this section
- OpenAI's Model Spec (version of August 18, 2026) says the assistant should weigh the cost of incorrect assumptions against the inconvenience of asking, fill information gaps from context or trusted sources before asking the user, and attempt the task unless a wrong assumption would cost too much. Source
- Anthropic's agent-eval guidance (January 9, 2026) says to test both the cases where a behavior should occur and where it should not, and that one-sided evals create one-sided optimization. Source
- The OpenAI Agents SDK pauses a run until a person approves or rejects tool calls marked as needing approval, always or by a per-call rule. Source
- tau-bench's entity-capture benchmark has 200 tasks with one missing field each and runs agent-directed and scaffolded strategies on the same task values, database and reward contract. Source
- QuestBench found model accuracy of only 40 to 50 percent on Logic-Q and Planning-Q, even where models could solve the fully specified problems. Source
- The MCP specification defines accept, decline and cancel responses to elicitation and forbids form-mode requests for passwords, API keys, access tokens or payment credentials. Source
Frequently asked questions
What should a clarification utility benchmark specification include?
A task record with the fully specified request, the withheld fact and the record holding it, the requester simulator's allowed knowledge, a decoy variant, tool state, the gold end state, accepted question targets and a turn budget, plus metric definitions, baselines, holdouts and a pre-registration plan.
How do you know a clarification score is not just memorized company conventions?
Trust only a result that holds when whole companies and at least one task family are excluded from development. If performance drops sharply on those holdouts, the agent learned the conventions of the companies it saw, not when and what to ask.
What sample, baseline and uncertainty would make a clarification result defensible?
Enough cases per family to detect a difference fixed in advance, never-ask and always-ask baselines run on the same cases, and binomial intervals with paired comparisons resampled by company. Families too small to meet the target should be reported as exploratory.
Why not just reward agents for asking clarifying questions?
Because asking has a cost. An agent that asks a generic question every time would score well on asking and badly on the requester's time. The decoy variants and the unnecessary ask rate penalize questions about facts the agent could have found itself.
Are there any measured scores for this clarification benchmark yet?
No. This page is a protocol for measuring clarification behavior on business tasks. It reports no scores, and any numbers in the design are parameters to be fixed before a run.
Sources
- QuestBench formalizes underspecified reasoning tasks as constraint satisfaction problems with one missing variable assignment and asks models to select the clarification question that acquires it. Source
- ClariQ, collected for the ConvAI3 challenge, labels clarification need from 1 to 4 and asks systems to select a clarifying question from a question bank or choose not to ask. Source
- AmbigQA annotates ambiguous open-domain questions whose acceptable outputs can be a single answer or multiple disambiguated question-answer pairs. Source
- Rao and Daume III's ACL 2018 work ranks clarification questions for Stack Exchange posts using a neural expected value of perfect information. Source
- tau-bench emulates conversations between a language-model-simulated user and an agent given domain-specific API tools and policy guidelines. Source
- The tau-bench paper proposes pass^k, the probability that all k independent trials of a task succeed, to measure reliability. Source
- NIST's Dataplot reference documents several confidence interval methods for a single proportion, including Wilson, adjusted Wald, Jeffreys and exact. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.