AI uses for records
Enterprise exception coverage benchmark: map the cases your evals skip
By SourceX Editorial · Updated
9 sections
Short answer
An enterprise exception coverage benchmark compares the exceptions a buyer's AI task requires with the cases a candidate dataset actually holds, workflow by workflow. Define four classes from documented workflows (rare decisions, policy conflicts, failed actions and recoveries), count each against a population denominator and a floor set in advance, keep observed prevalence apart from oversampling, and turn every unfilled cell into a sourcing brief. This page publishes the protocol, not results.
Evaluation split
Enterprise exception
Key takeaways
- Exception coverage is a property of the data, measured before any model is scored.
- Four classes defined from documented workflows (rare decisions, policy conflicts, failed actions and recoveries) work across finance, procurement, field service and IT.
- Public agent suites cover a handful of shared domains; a buyer's own exception mix has to come from its workflows or licensed records.
- An empty cell has three causes, absent from the population, excluded for rights, or never sampled, and each needs a different fix.
- Report observed prevalence and oversampled evaluation counts side by side, never one as the other.
What does an enterprise exception coverage benchmark measure?
It measures whether the exceptions a buyer's AI task must handle are present, in enough complete cases, in the data the buyer could license or test on. It scores data, not models: every cell of a workflow-by-exception-class grid gets a population count, a candidate-set count, a floor fixed in advance and a status.
Exceptions deserve their own benchmark because they are where policy, judgment and failure paths concentrate, and where routine samples run thin. Anthropic's January 2026 guidance on evaluating AI agents recommends building early suites from real failures found in the bug tracker and support queue, and building balanced problem sets that test both where a behavior should occur and where it should not, warning that one-sided evals create one-sided optimization. A sample of ordinary work rarely contains enough of either side for the cases that matter.
This page publishes the benchmark design as a research commission. SourceX has not run it and no coverage counts exist; a future report would preregister the method, obtain rights and disclosure approvals from participating companies, and publish negative results, including categories nobody could supply. The help-desk version of a coverage matrix already has its own article; this protocol spans workflows and starts from what the buyer needs.
Why public agent benchmarks leave most enterprise exceptions out
Public agent benchmarks are built so that many models can be compared on the same tasks, so they concentrate on a few shared domains rather than any one company's exceptions. That makes them a weak proxy for whether a specific enterprise task is covered.
The current suites show the pattern. The τ-bench repository from Sierra describes a simulation framework for customer-service agents whose released text benchmark uses the same 50 frozen tasks per domain across airline, retail and telecom, with banking and intake domains also available. ServiceNow's WorkArena-L1 holds 19,912 instances drawn from 33 atomic tasks on the ServiceNow interface, and WorkArena++ composes those components into 682 tasks. TheAgentCompany lists 175 task images that run against self-hosted workplace services such as GitLab, Plane, ownCloud and RocketChat, with roles from software engineer to financial staff and administrator.
These are valuable, carefully built suites, and none claims to represent the exceptions in a particular company's order-to-cash, month-end close or claims workflow. They could not: policies, approval thresholds and failure paths differ by company. Anthropic's September 2025 Economic Index report adds a demand-side reason to care, finding that tasks requiring dispersed context or complex regulatory navigation lag in adoption and that complex deployments may be constrained more by access to contextual information than by model capability. Exceptions are where that context is thickest.
Four exception classes, defined from documented workflows
Define the classes from the documented workflow first and look for evidence in records second, never the reverse. A case counts as an exception only if it departs from a documented rule or path, so every candidate carries a pointer to the rule it departs from and to the record that shows the departure.
Four classes cover most enterprise workflows, and they are deliberately about what happened rather than which system holds it, so the same grid works for finance, procurement, field service and IT. The examples in the table are illustrative, not descriptions of any vendor's documented behavior.
Process mining gives a reproducible way to find candidates at scale. PM4Py's alignment-based conformance checking aligns each case's event trace with a process model and classifies a move on log, an event the model does not allow, as a deviation; deviations that recur become candidate exception types for a process owner to confirm, merge or reject. Where no process model exists, counting distinct trace variants is a weaker first pass that still exposes the long tail.
| Class | Departs from | Evidence in the records | Illustrative example |
|---|---|---|---|
| Rare decision | A valid but infrequent branch of a documented rule | Decision or approval field set to the rare value, with the approver's role | A purchase order above a delegated limit routed to the CFO for approval |
| Policy conflict | Two rules that disagree, or a rule overridden with approval | Override flag, exception code or approval note that cites the rule | A refund outside the return window approved by a team lead |
| Failed action | An expected system action that errored or was rejected | Error status, rejection code or returned transaction | A supplier payment returned by the bank, or a change request rolled back |
| Recovery | The steps after a failure that restore the intended outcome | Reversal, retry, credit or escalation linked to the failed record | A missed field-service appointment rescheduled with a credit issued |
The Enterprise Exception Coverage Atlas: one grid per buyer task
The atlas is a grid with one row per workflow and one column per exception class, drawn once for the buyer's requirement and once for every candidate dataset. Each cell holds five facts and one status, so two grids can be laid over each other and compared cell by cell.
The five facts are the buyer's floor, meaning the minimum number of complete cases set before anyone counts; the population count for a stated period, with its denominator; the candidate-set count; how many of those have a settled outcome and a known rule version; and a sampling note saying whether cases were drawn at observed rates or oversampled, with inclusion probabilities.
The grid below is an Illustrative status layer for a buyer building a back-office agent. The statuses are examples chosen to show every rule in the next section, not measurements of any company or industry.
Read it by row and by column. A row full of thin and missing cells means one workflow is under-supplied and the fix is often a different supplier; a column full of them means the class itself is hard to capture, which is worth knowing before paying for workflow-specific sourcing.
| Workflow (Illustrative) | Rare decisions | Policy conflicts | Failed actions | Recoveries |
|---|---|---|---|---|
| Order-to-cash | Covered | Thin | Covered | Missing |
| Procure-to-pay | Thin | Missing | Covered | Thin |
| Month-end close | Covered | Unmeasurable | Thin | Missing |
| Field service dispatch | Covered | Thin | Thin | Excluded |
| IT service management | Covered | Covered | Covered | Thin |
| Insurance claims | Thin | Absent in population | Missing | Missing |
Covered, thin, missing or absent: a rule for every cell
Give every cell exactly one status from written rules, and keep three kinds of empty cell apart: absent from the population, excluded for rights or privacy, and present but never drawn into the candidate set. They look identical in a sample and need opposite fixes.
Set floors before counting, from the smallest performance gap the buyer needs to detect. Anthropic's Adding Error Bars to Evals (November 2024) recommends power analysis to work out how many questions an eval needs to detect a difference of a given size, and the same logic sets a cell's floor. The paper also recommends clustering standard errors when questions are related; when several tasks are cut from one long case, count the case, not the tasks, toward the floor.
| Status | Rule | What the buyer does next |
|---|---|---|
| Covered | Candidate count at or above the floor, outcomes settled, rule version known | Use the cell and report its count beside every score |
| Thin | Above zero but below the floor | Treat cell results as indicative and source more cases |
| Missing | Present in the population, absent from the candidate set | Widen the candidate scope or add a supplier |
| Absent in population | No cases in the population for the period | Treat as a different source problem, not a sampling gap |
| Excluded | Present but removed for rights, privacy or contract reasons | Report how many were excluded and why; seek a permitted source |
| Unmeasurable | The signal was not recorded for part of the period | Mark the period and never count the gap as zero |
Observed prevalence versus deliberate oversampling
Report two numbers for every cell and never let one stand in for the other: how often the exception occurs in the population, and how many cases the evaluation set holds after oversampling. The first describes the business; the second buys statistical power.
Oversampling is legitimate and usually necessary, because a class that makes up a tiny share of a workflow would otherwise contribute a handful of cases. Record each case's inclusion probability so an overall score can be reweighted to real-world frequency when someone needs a population figure, and publish per-cell results unweighted so a buyer sees performance where it matters. The arithmetic of allocating a fixed budget across slices is covered in the stratified evaluation guide linked below.
Balance matters inside a cell too. Anthropic's guidance warns against class-imbalanced evals and recommends testing cases where a behavior should occur alongside cases where it should not. For exceptions, that means pairing each policy-conflict case with a close routine case where an override would be wrong, so an agent that escalates everything does not score as an exception expert.
Comparing buyer-required coverage with the licensed candidate set
The comparison is a cell-by-cell difference between two grids: the buyer's required floors and the candidate set's covered counts. Every required cell that is not covered becomes a gap with a named cause.
Write the requirement before looking at candidate data. A requirement grid lists the workflows the task touches, the classes it must handle, the floor per cell and the acceptance test, such as reaching the correct end state on policy-conflict cases at a rate the buyer names. Anthropic's guidance distinguishes the outcome, the final state in the environment, from what the agent says it did, which is why every required cell should name the end-state evidence a case must carry.
Then compare. Illustrative: a buyer building a procure-to-pay agent requires policy-conflict and recovery cases. A candidate supplier's records cover failed actions well, but approvals outside policy were recorded in email rather than the ERP, so the policy-conflict cell is unmeasurable until those approvals can be linked to their purchase orders. That is a supply problem with a specific fix, not evidence that the exceptions never happened.
Keep the denominators visible throughout. Population counts come from the supplier's systems for a stated period, candidate counts from the sampled or delivered set, and each number carries its date, extraction version and the rule set it was classified under.
From unfilled cells to a sourcing brief, without invented demand
An unfilled cell becomes a sourcing brief that states what evidence each case must carry, not how many buyers want it. The brief describes a requirement; it makes no claim about market demand, available stock or acceptance.
SourceX fits at this step. Nothing is sold from stock: SourceX sources datasets from U.S. businesses against a buyer's request, and the request-data form is where buyers set out scope, volume, language, format, timing and licensing needs; a requirement grid and its gaps are a precise way to state that scope. On the supply side, no data is delivered until the company has accepted the scope and terms and signed the agreement, so a cell excluded for rights stays excluded unless the company agrees otherwise.
A complete brief for one cell contains:
- The workflow and exception class, with the documented rule the cases depart from.
- The floor and the counting unit: one source case, not each task derived from it.
- The evidence each case needs: trigger, rule version in force, approval with the approver's role, actions taken and settled outcome.
- The systems where that evidence usually sits and the linkage needed to keep it together in one case.
- Exclusions: personal data, third-party content and contract restrictions that would remove cases.
- The acceptance test the cases will be held to, and whether they will be drawn at observed rates or oversampled.
What this protocol cannot show, and what has not been measured
This page is a research commission, not a report. SourceX has not measured exception coverage in any enterprise sample, and no status or count on this page is a finding.
Before any result is published, the study needs a preregistered taxonomy and floors, an approved cross-workflow inventory from each participating company, dated population counts with denominators, inter-rater agreement on class assignment, rights and disclosure approval for every published count, and a commitment to report missing, excluded and negative results.
Even then the design has limits. Coverage counts show whether cases exist, not whether a model trained or tested on them improves. Process models capture documented paths, so discretionary practice nobody wrote down needs a process owner to state the rule. One company's exception mix does not represent an industry.
Methods note: the four classes, the atlas grid, the status rules and the brief template are editorial frameworks built on 2026-10-26. They draw on Anthropic's agent-evaluation guidance (January 2026) and statistical recommendations (November 2024), the τ-bench, WorkArena and TheAgentCompany repositories, PM4Py's conformance-checking documentation and Anthropic's September 2025 Economic Index report. Not measured: exception prevalence in any workflow, coverage of any real dataset, and the effect of coverage on model results.
Frequently asked questions
What counts as an exception in enterprise AI evaluation?
A case that departs from a documented rule or path: a rare but valid decision, a conflict between rules or an approved override, a system action that failed, or the recovery after a failure. Routine cases with unusual wording are not exceptions. Tie every case to the rule it departs from.
How many cases does each exception cell need?
Set the floor before counting, from the smallest performance gap the buyer needs to detect, using a power calculation. Count source cases rather than tasks derived from one case, because related tasks carry less information than independent ones.
Should an evaluation set match real-world exception frequency?
Not for measuring each class. Oversample so every class has enough cases, record inclusion probabilities, and reweight to observed prevalence only when you need a population-level figure. Report both numbers and label which is which.
How is this different from a support ticket coverage matrix?
A support coverage matrix checks one help desk's sample against its own case population. This protocol spans workflows, starts from the buyer's required cells and separates three causes of an empty cell, so one requirement can be compared against several candidate suppliers.
Has SourceX published exception coverage results?
No. This page publishes the protocol only. Any results would follow a preregistered design with rights and disclosure approval, and would include missing, excluded and negative findings.
Sources
- Anthropic's January 2026 guidance on agent evaluations recommends starting from real failures in the bug tracker and support queue, building balanced problem sets that test where a behavior should and should not occur, avoiding class-imbalanced evals, and grading the outcome as the final state in the environment. Source
- Anthropic's Adding Error Bars to Evals (November 2024) recommends clustering standard errors when questions are related and using power analysis to decide how many questions an eval needs to detect a difference of a given size. Source
- The τ-bench repository describes a simulation framework for customer-service agents whose released text benchmark uses the same 50 frozen tasks per domain across airline, retail and telecom, with banking_knowledge and intake domains also available. Source
- WorkArena-L1 includes 19,912 unique instances drawn from 33 atomic tasks on the ServiceNow interface, and WorkArena++ contains 682 tasks composed from those components. Source
- TheAgentCompany benchmarks agents on professional tasks against self-hosted services including GitLab, Plane, ownCloud and RocketChat, lists 175 task images and covers roles including software engineer, product manager, data scientist, human resources, financial staff and administrator. Source
- PM4Py's alignment-based conformance checking aligns each trace with a process model and classifies a move on log, an event not mirrored in the model, as a deviation. Source
- Anthropic's September 2025 Economic Index report finds that tasks requiring dispersed context or complex regulatory navigation are further behind in adoption and that complex AI deployment may be constrained more by access to information than by model capability. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.