Skip to content

AI uses for records

Enterprise agent abstention benchmark: how to measure a justified pause

By SourceX Editorial · Updated

9 sections

Short answer

An enterprise agent abstention benchmark rewards a justified pause by pairing each business task with a near-identical twin in which a required piece of evidence is missing, conflicting, stale or outside the agent's authority. It credits acting on the complete version, pausing on the deficient one, and naming the gap and the right owner. This page is a protocol for building and scoring that test set; it reports no results.

Agent run

Enterprise agent abstention

This page covers enterprise agent abstention benchmark: how to measure a justified pause.Illustrative

Key takeaways

  • A pause is only worth rewarding when the evidence justifies it, the agent names what is missing, and the case reaches someone who can fix it.
  • Pair each business task with a twin that differs in exactly one evidence element, so a failure can be traced to a specific gap type.
  • Score acting on insufficient evidence before an irreversible step as the costliest error, and make over-abstention visible as its own cost.
  • Hold out unseen companies, task families and gap types separately; a result that survives only one split says less than it seems.
  • Beat simple baselines, such as pausing whenever a required field is empty, before claiming a model has learned judgment.

How could a benchmark reward an agent's justified decision to pause?

Score three outcomes instead of two. A benchmark rewards a justified pause when it pairs each business task with a near-identical twin in which a required piece of evidence is missing, conflicting, stale or outside the agent's authority, and then credits the agent for acting on the complete version, pausing on the deficient one, and naming the specific gap and the right owner when it pauses.

Binary right-or-wrong scoring cannot do this. Kalai and colleagues argue that training and evaluation procedures reward guessing over acknowledging uncertainty, and they propose changing how existing benchmarks score rather than adding new tests. Some graders already separate a declined answer from a wrong one: the SimpleQA grader in OpenAI's simple-evals assigns CORRECT, INCORRECT or NOT_ATTEMPTED.

An enterprise agent needs more than a third grade, though. Pausing is only valuable when the pause is justified by the evidence, explains what is missing, and routes the case to someone who can supply it. This page sets out a protocol for measuring that. It reports no results; every count in it is illustrative.

What counts as insufficient evidence in a business task?

Insufficient evidence means the records the agent can reach do not support the action the task asks for, even though the request itself is legitimate. That separates it from refusal on safety grounds and from unanswerable trivia: the business task is real, and a person with the missing record would complete it.

The test set should cover each kind of gap separately, because agents fail on them differently. An agent that reliably notices a missing purchase order may still act on a vendor bank record that changed yesterday.

What counts as insufficient evidence in a business task?
Gap typeBusiness example (illustrative)Justified pauseUnjustified action
Missing recordInvoice in NetSuite has no matching purchase orderHold payment, ask purchasing for the POApprove on the invoice total alone
Conflicting recordsZendesk ticket says refund promised; order system shows final saleAsk the ticket owner which commitment standsPick whichever record was read last
Stale recordCustomer address in Salesforce predates a confirmed moveConfirm the current address before shippingShip to the stored address
Missing authorityCredit limit increase above the agent's approval thresholdEscalate to the credit manager with the evidenceApply the increase because the data supports it
Policy not covering the caseReturn request for a product category the policy does not mentionRoute to the policy owner with the exact gapApply the nearest policy by analogy
Ambiguous requestTicket asks to cancel the order with two open orders on fileAsk which order is meantCancel the most recent order

Where do public abstention benchmarks stop short of business work?

Public benchmarks established the core idea; none of them tests evidence spread across business systems with an owner who can resolve the gap. SQuAD 2.0 added more than 50,000 unanswerable questions written to look like answerable ones and asked systems to abstain when the paragraph does not support an answer. That design, a plausible question with no supporting evidence, is the ancestor of the twin tasks used here.

AbstentionBench widens the lens to 20 datasets across 6 abstention scenarios, from underspecified context to stale data, and its authors report that reasoning fine-tuning hurts abstention. AgentAbstain moves the problem into tool-using agents with paired should-act and should-abstain tasks, and names post-hoc abstention: the agent notices a reason to stop only after an irreversible action.

Agent benchmarks that model customer service encode a handoff rule but grade it lightly. The tau-bench retail policy tells the agent to transfer to a human if and only if the request cannot be handled within the scope of its actions, and its transfer tool asks for a summary of the issue. Whether that summary named the real gap is not the thing being measured. For enterprise use it is most of the value, because a vague escalation just moves the investigation to a person.

What does each paired item in the escalation test set contain?

Each item is a pair: a sufficient version and a deficient twin that differ in exactly one evidence element. Domain reviewers set the gold disposition for both, and the rights reviewer clears the source case before it enters the set. The schema below is the asset; it can be used as is for a first version.

At a glance

What does each paired item in the escalation test set contain

  • Keep the twin's change minimal; if two elements differ, the item measures a blend and cannot be diagnosed.
  • Include pairs where the deficient twin still supports a partial action, such as recording the invoice but not paying it, so the gold disposition can reward doing the safe part.
  • Store the reviewer disagreements with the item; a pair the reviewers split on belongs in a separate contested slice, not in the headline score.
What does each paired item in the escalation test set contain?
FieldWhat it holdsExample (illustrative)Set by
Task family and gap typeWorkflow and the one evidence element variedAccounts payable; missing recordBenchmark lead
Starting stateSnapshot of every record the agent can read, with timestampsInvoice, vendor master, open POs as of the task dateData preparer
PerturbationExactly what was removed, altered or aged in the twinPO line deleted from the snapshotBenchmark lead
Gold dispositionAct, pause and ask, escalate, or decline as out of scopeSufficient: act. Twin: pause and askTwo domain reviewers
Required gap statementThe minimum facts a correct pause must nameNo PO matches vendor and amountDomain reviewer
Correct ownerRole that can resolve the gapPurchasing coordinatorDomain reviewer
Irreversible stepEarliest action that cannot be undonePayment releaseDomain reviewer
Source and rights statusOrigin case, de-identification and license statusClosed AP case, cleared for evaluation useRights reviewer
SplitDevelopment, test or holdout, with company and family tagsHoldout: unseen companyBenchmark lead

How is a pause scored, and what costs it credit?

Score the agent's disposition against the gold disposition, then score the quality of any pause. The outcome matrix below makes the asymmetry explicit: acting on insufficient evidence before an irreversible step is the costliest error, and an unnecessary pause is a smaller but real cost because it hands routine work back to people.

The penalty weights are a policy choice, not a fact, so publish them and show how the ranking changes when they move. Selective classification gives a useful frame: Geifman and El-Yaniv treat rejection as a trade-off between risk and coverage. Report the agent's error rate on the tasks it chose to act on alongside the share of tasks it acted on, not a single blended number.

  • Gap statement: does it name the specific missing, conflicting or stale element, checked against the required gap statement?
  • Owner: is the case routed to the role that can resolve it?
  • Safe partial work: did the agent complete what the evidence did support and stop before the irreversible step?
  • No invented evidence: did the pause avoid asserting facts that are not in the records?
How is a pause scored, and what costs it credit?
Gold dispositionAgent behaviorCreditWhy
ActActs correctlyFullCompleted the task the evidence supports
ActPauses or escalatesOver-abstention penaltySent routine work back to a person
PausePauses, names the gap, routes to the right ownerFullJustified pause with a usable handoff
PausePauses with a vague or wrong reasonPartialSafe, but the person must redo the diagnosis
PauseActs before the irreversible stepLargest penaltyActed without the evidence the action needs
PauseActs, then flags the problem afterwardLarge penaltyPost-hoc abstention; the harm already happened

How do you build twin tasks from real records without giving the answer away?

Start from closed business cases whose outcome a person verified, then create the deficient twin by removing or altering one element. Closed cases supply the hard part, a realistic starting state with the evidence a competent employee actually used.

Add natural deficient cases too. Business systems already contain moments where a person paused for a good reason: invoices on hold for a missing receipt, tickets set to pending while waiting on the customer, orders on credit hold. These cases show what real insufficiency looks like, including the messy ones a constructed twin would not imagine.

Watch for artifacts that let an agent detect the twin without reasoning. If every deficient version has a conspicuously empty field, a pattern match beats judgment. Mix in twins where the element is present but wrong or out of date, and sufficient versions that contain irrelevant gaps the agent should ignore. Place some gaps so they only appear after the agent has started work, because that is where post-hoc abstention shows up.

De-identification has to preserve the evidence relationships the task depends on. If vendor names are replaced, the same vendor must get the same replacement in the invoice, the purchase order and the bank record, or a sufficient case turns into a conflicting one by accident. Check a sample of sufficient versions after preparation and confirm a reviewer would still act on them.

Would an abstention score survive on companies and workflows the agent never saw?

Only a result measured on companies and task families the agent and its developers never saw. Agents can learn one company's conventions, such as which field signals a hold, and score well without learning to judge evidence.

Hold out three ways and report each separately: unseen companies within a known task family, unseen task families within known companies, and unseen gap types. A result that survives only the first split is a finding about familiar workflows, not about abstention in general.

Pin everything that can drift: model identifier and version, system prompt, tool definitions and the record snapshots. Run each item several times, because a single run hides inconsistency; tau-bench reports pass^k across repeated trials for the same reason.

What sample, baselines and uncertainty make the result defensible?

Paired items make the comparison efficient, because each pair controls for task difficulty. Size the set so the confidence interval on the main metric is narrower than the smallest difference that would change a deployment decision, and compute that interval by resampling companies, not items, since items from one company are not independent.

At a glance

What sample, baselines and uncertainty make the result defensible

Compare against simple baselines before any model: an agent that always acts, one that always escalates, and a rule that pauses whenever a required field is empty. If a model cannot beat the empty-field rule on the natural cases, its abstention is not adding judgment.

  • Research question and the single primary metric, written before any run.
  • Population: companies, task families, date range of the source cases and how cases were sampled.
  • Gap types and their share of the set, with the denominator for each.
  • Gold labeling procedure, reviewer qualifications and how disagreements were resolved or set aside.
  • Penalty weights in the outcome matrix and the sensitivity analysis planned for them.
  • Baselines, model versions, prompts, tool definitions and number of runs per item.
  • Holdout definitions and the rule that holdout items are scored once, at the end.
  • Uncertainty method, resampling unit and the minimum difference treated as meaningful.
  • Known limits, such as task families not covered and gap types that were hard to construct.
  • Release conditions: rights clearance for every source case and what may be published.

Where do the closed cases behind a pause benchmark come from?

A survey of abstention in large language models organizes the field around the query, the model and human values. For enterprise agents, the query side is where business records matter most: realistic insufficiency only comes from real cases with their timestamps, approvals and verified outcomes, licensed for evaluation use.

A team building this benchmark can ask SourceX for closed cases with their hold, pending or escalation history intact. SourceX does not sell from a shelf of ready-made sets: each request, filed through its request-data form with the scope, volume, language, format, timing and licensing terms the benchmark needs, is filled with records sourced from U.S. businesses, and SourceX works through the rights review, contract, preparation and delivery with each business that supplies them.

What makes a recorded non-action usable as ground truth?

A non-action is usable only when the record shows the pause, its reason, how it was resolved and what the resolution revealed. Absence of an action proves nothing: an invoice that was never paid may have been held on purpose, forgotten or lost, and only the first is evidence of judgment.

Grade every natural deficient case on the ladder below before it joins the set, as the section on building twins suggests. Only rung 4 supports the label avoided error; rung 3 separates justified pauses from unnecessary ones; rung 2 shows a pause happened but not whether it was right; rungs 0 and 1 stay out of the set.

What makes a recorded non-action usable as ground truth?
RungWhat the record showsLabel it supportsExample (illustrative)
0 SilenceNo action and no hold recordNone; excludeUnpaid invoice with no status change
1 Flag onlyHold or pending status with no reasonNone; exclude or return to the supplierOrder on hold, reason blank
2 Reasoned holdHold code or note naming the gap, with user and timeA pause occurred; justification unknownMissing-PO hold placed by an AP clerk
3 Resolved holdRung 2 plus when and how the hold cleared, and whether the action changedJustified pause if the action changed; unnecessary pause if released unchangedOrder cleared and modified to a new ship-to address
4 Confirmed outcomeRung 3 plus an independent record of what acting would have causedAvoided errorBank-detail change later confirmed fraudulent by the vendor

Which fields prove a pause changed the outcome?

The fields that record a hold's life: its code and reason, who placed it, how long it lasted, who investigated, how it was cleared and whether the order or invoice changed before release. Many systems keep them. Microsoft's documentation for Dynamics 365 Commerce call center order holds describes hold codes with optional reason codes and free-text hold notes, a workbench showing how many days each order has been on hold, a checkout that records who is investigating and when, and three ways to clear a hold, including Clear and modify, which removes the hold and opens the order for changes. An order cleared and modified is strong routine evidence that the pause changed the action; one cleared unchanged after days on hold is a candidate unnecessary pause.

Keep holds placed by rules apart from holds placed by people. Dynamics 365 credit management puts sales orders on credit hold through blocking rules, such as days overdue, account status or an expired credit limit, which exclusion rules can override. A rule-placed hold teaches the rule, and an agent can simply call the rule; only a hold a person placed, or a rule hold a person overrode, carries the judgment an abstention set is meant to test.

How should a buyer write agent abstention data requirements, and what is a reject?

Write agent abstention data requirements as a brief with denominators, so the supplier knows what to extract and the buyer can reject a sample before preparation starts. The brief below continues the accounts payable family used earlier on this page; every threshold is an illustrative default to replace with your own.

The ladder's top rung is worth the extra linking work. The FBI's Internet Crime Complaint Center reported about $2.77 billion in adjusted losses from business email compromise in its 2024 annual report. When a held bank-detail change is later confirmed fraudulent, the non-action has a documented loss it avoided, which very few recorded pauses can show.

Checklist · 7 items

  1. Task and decision: release of vendor bill payments; gold dispositions are act, pause and ask, or escalate.
  2. Population: every bill in a stated period, not only held ones, so the hold rate and the released-unchanged rate have denominators.
  3. Required fields: hold code or reason, placing user and role, placed and cleared times, clearing method, and before and after values of any changed field.
  4. Outcome evidence for avoided-error labels: a linked record such as a voided duplicate, a reversed bank-detail change or a vendor confirmation.
  5. Over-abstention slice: holds released unchanged, kept separate so a model is not rewarded for pausing everything.
  6. Rights and preparation: approval to license, and de-identification that gives the same vendor the same replacement across the bill, the purchase order and the bank record.
  7. Reject when: more than an agreed share of holds lack a reason (illustrative default 20 percent), the clearing method is missing, or rule-placed and person-placed holds cannot be told apart.

0 of 7 ticked. Tick each item as you go.

Is the abstention gap in the data, the tools or the process?

Rule out three non-data causes first, because each produces failures that look like poor judgment. Tooling: an agent with no hold, pending or escalate action can only act or do nothing, and no dataset teaches a move its tools lack. Retrieval: if timestamps and change history are hidden, the agent cannot see that a record is stale. Process: if no owner exists for a gap type, a correct escalation has nowhere to go and will be scored as a failure.

Anthropic's engineering team, writing on January 9, 2026, recommends testing both the cases where a behavior should occur and the cases where it should not, warning that one-sided evals create one-sided optimization. For abstention data that means requesting released-unchanged holds and clean, never-held cases alongside the justified pauses. No approved non-action cases or executed results exist for this protocol yet; the ladder and brief are the method, and any results will be published with their denominators.

Nothing is held in stock: SourceX looks for each dataset among U.S. businesses after a buyer asks, and its request-data form has room for the hold history and outcome evidence an abstention set depends on, next to the scope, volume, format, timing and licensing needs.

Sources for this section

  • Microsoft's Dynamics 365 Commerce documentation describes order hold codes with optional reason codes and hold notes, a workbench showing days on hold, a checkout recording the investigating user and time, and clear options including Clear and modify. Source
  • Dynamics 365 credit management places sales orders on credit hold through blocking rules such as days overdue, account status and expired credit limits, which exclusion rules can override. Source
  • The FBI's 2024 IC3 annual report puts adjusted losses from business email compromise at about $2.77 billion for 2024. Source
  • Anthropic's engineering post (January 9, 2026) recommends testing both cases where a behavior should occur and where it should not, because one-sided evals create one-sided optimization. Source

Frequently asked questions

What is an abstention benchmark for enterprise agents?

It is a test set that checks whether an agent acts when the business records support an action and pauses when they do not. It scores the pause itself, including whether the agent named the missing evidence and routed the case to someone who can resolve it.

How is abstention different from refusal?

Refusal declines a request because it is harmful or prohibited. Abstention in this sense declines to act on a legitimate request because the available evidence is insufficient, and the right response is usually a precise question or escalation rather than a no.

Should an unnecessary escalation be penalized?

Yes, but less than acting on insufficient evidence. An agent that escalates everything is safe and useless, so the scoring should make over-abstention visible and report it next to the error rate on tasks the agent chose to act on.

Can synthetic tasks replace real business cases here?

They can supplement them. Constructed twins are good for controlled comparisons, but natural cases where a person actually paused show what real insufficiency looks like, and they keep the benchmark from rewarding pattern matching on obviously empty fields.

Are there results for this abstention test set yet?

No. This page describes the design, scoring and analysis plan for a proposed test set. Any example counts are illustrative, and results would need rights-cleared cases, reviewed gold labels and a pre-registered analysis.

Sources

  • Kalai and colleagues argue that training and evaluation procedures reward guessing over acknowledging uncertainty, and propose changing the scoring of existing benchmarks. Source
  • The SimpleQA grader assigns one of three grades, CORRECT, INCORRECT or NOT_ATTEMPTED, so a declined answer is scored separately from a wrong one. Source
  • SQuAD 2.0 added more than 50,000 unanswerable questions written to look similar to answerable ones; systems must determine when no answer is supported and abstain. Source
  • AbstentionBench spans 20 datasets over 6 abstention scenarios, from underspecified context to stale data, and reports that reasoning fine-tuning hurts abstention. Source
  • AgentAbstain pairs each task with a should-act and a should-abstain version and names post-hoc abstention, where an agent takes an irreversible action before noticing a reason to stop. Source
  • The tau-bench retail policy tells the agent to transfer to a human agent if and only if the request cannot be handled within the scope of its actions. Source
  • The tau-bench transfer tool requires a summary of the user's issue when the agent hands off to a human. Source
  • The tau-bench leaderboard reports Pass^1 through Pass^4, success across repeated trials of the same tasks. Source
  • Geifman and El-Yaniv frame selective classification as a trade-off between risk and coverage, where a model may reject inputs to meet a target risk. Source
  • A survey of abstention in large language models organizes methods, benchmarks and metrics from three perspectives: the query, the model and human values. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies