AI uses for records
Business exception generalization benchmark: a holdout protocol
By SourceX Editorial · Updated
10 sections
Short answer
This is a protocol, not a result. To build a business exception generalization benchmark, group resolved exceptions into families by trigger and policy, hold out every case of one family from training, prompts and model selection, and test on it with and without its written policy alongside unseen cases from seen families. Rotate the held-out family and treat families as the unit of uncertainty. No results are reported.
Evaluation split
Business exception
Key takeaways
- A random split lets an agent recognize familiar exception families, so it cannot show generalization to a new type.
- Define families from the raise-time record by trigger and policy, and list each family's nearest sibling.
- Compare seen-family cases, the held-out family with its policy, and the held-out family without it.
- Select models on a separate validation family; any tuning on the test family makes the result an upper bound.
- Treat the family as the unit of uncertainty and report per-family results, not only a pooled score.
How do you hold out an entire exception type to test agent generalization?
Group resolved business exceptions into families that share a trigger and a resolution policy, then remove every case of one family from anything the agent or its developers see: training data, prompts, examples and model selection. Test on that family alongside unseen cases from the families that were seen, rotate which family is held out, and treat the family, not the case, as the unit of uncertainty.
This page is a protocol. It sets out the question, population, split, variables, comparisons, metrics, uncertainty handling and limits for a business exception generalization benchmark, and it reports no results. Any numbers below are design parameters or labelled illustrations.
Why can't a random split show that an agent generalizes?
Because a random split puts cases from every exception family on both sides. An agent can then score well by recognizing the family and replaying how similar cases were resolved, which measures recall of familiar patterns, not the ability to handle an exception it has never met. In production, the costly failures come from the new kind of exception: a supplier sends a credit memo in a format nobody has seen, or a policy creates a new category of refund.
Grouped splitting is the standard remedy. scikit-learn's LeaveOneGroupOut, for instance, builds each split by holding out all samples that belong to one group, given an array that assigns every sample to a group. Here the group is the exception family.
The closest research precedent is compositional generalization. The SCAN benchmark's add-primitive split trains on every command that does not use jump, plus the bare jump command alone, and tests on all compositional commands that use jump. That is the shape this protocol borrows: the agent may know that an exception family exists, through its written policy, without having seen a single worked case of it.
Existing agent benchmarks are organized in ways that make this possible but do not usually do it. tau2-bench groups tasks by domain, such as airline, retail and telecom, which is a coarser holdout than an exception family. WorkArena distinguishes atomic tasks from WorkArena++ tasks that compose those atomic components into longer use cases, which tests composition rather than an unseen exception class.
What counts as an exception family?
A family is a set of cases that share a trigger condition and a governing resolution policy. Define families from the record as it stood when the exception was raised, not from how it was resolved, otherwise the family label leaks the answer.
Granularity decides what the benchmark measures. Families that are too broad, such as all invoice problems, leave nothing new in the held-out set. Families that are too narrow, such as price variance on one product line, leave a near-identical sibling in training. Record each family's nearest sibling and check how much of the held-out family's resolution logic the sibling already teaches. The table shows illustrative families from accounts payable and order operations.
| Illustrative family | Trigger | Governing policy | Nearest sibling to watch |
|---|---|---|---|
| Price variance | Invoice unit price differs from the PO price beyond tolerance | Price tolerance and approval policy | Freight or surcharge variance |
| Quantity mismatch | Invoiced quantity exceeds received quantity | Receipt matching policy | Missing receipt |
| Duplicate invoice | Same supplier, amount and reference seen before | Duplicate payment control | Credit memo applied twice |
| Missing receipt | Invoice arrives before any goods receipt | Hold and follow-up policy | Quantity mismatch |
| Address change after shipment | Customer changes destination once the order has shipped | Carrier intercept and fee policy | Address change before shipment |
| Substitution request | Ordered item unavailable and a substitute is proposed | Substitution and customer consent rules | Partial shipment and backorder |
The task family split and benchmark reporting plan
This is the plan to register before any model run. Each row names a design component, the choice this protocol makes, the reason, and what the report must state about it.
At a glance
The task family split and benchmark reporting plan
Sampling follows the families, not the raw case stream. Draw cases within each family and company with a fixed random seed, cap any single company's share of a family so one firm's habits do not define it, and keep cases from the same underlying incident, such as several invoices from one disputed shipment, on the same side of every split. Record for each case the variables a reviewer will need to explain a failure: company, system, family, raise date, resolution, whether the resolution was escalated, whether more than one resolution was acceptable, and the policy version in force on the raise date.
Define the population window so that every held-out case is closed and its outcome settled. An exception resolved last week may still be reversed, and a reversed resolution makes the recorded ground truth wrong. Leave a settling period after the last case, state it, and exclude cases reopened inside it.
| Component | Design choice | Reason | What the report states |
|---|---|---|---|
| Research question | Does accuracy on an unseen exception family fall short of accuracy on unseen cases of seen families, and by how much? | Separates new-family generalization from case-level generalization | The question verbatim and the primary metric |
| Population | Closed, rights-cleared exception cases from several companies, each with its record at raise time and the recorded resolution | Ground truth must exist and be licensable for benchmark use | Companies, systems, collection dates, inclusion and exclusion rules |
| Family assignment | Two domain experts assign families from the raise-time record; disagreements adjudicated before splitting | Unstable labels make the holdout meaningless | Taxonomy version, agreement rate, adjudication rule |
| Split | Leave one family out, rotated over every family with enough cases; a separate validation family is held out from both training and test | Every family gets a turn and model selection never touches the test family | Families held out per fold and case counts per arm |
| Policy exposure | Held-out family tested with and without its written policy document | Distinguishes reading a new rule from inferring it | Exact policy text supplied in each condition |
| Baselines | Route-to-human rule, most frequent resolution, and the same agent with no tool access | Shows whether the agent beats trivial strategies on the new family | Each baseline's definition and version |
| Grader | End-state checks against the recorded resolution, with expert adjudication where several resolutions are valid | Business exceptions often allow more than one correct outcome | Grader version, check list and adjudication rate |
| Metrics | Resolution accuracy, correct escalation rate, forbidden action rate, reported per family | A wrong irreversible action is worse than a needless escalation | Per-family values, never only a pooled score |
| Uncertainty | Clustered intervals by family, repeated trials per case | Cases within a family are not independent | Number of families, cases and trials behind every interval |
Which comparisons does the design isolate?
Three arms per fold, each answering a separate question. Running only the held-out arm tells you a number with nothing to compare it to.
- Arm A, seen families, unseen cases: the in-distribution reference. Cases come from families present in training, but the cases themselves were never seen.
- Arm B, held-out family with its policy: the agent receives the written policy for the new family but no worked cases. The gap between A and B is the new-family generalization gap.
- Arm C, held-out family without its policy: the same cases with no policy text. The gap between B and C shows how much the written rule helps compared with inference from general knowledge.
- Optional dose arm: add one, two, four and then more worked cases from the held-out family, in the spirit of SCAN's few-shot conditions, which add increasing numbers of examples with several random samples at each size. This shows how quickly the gap closes, which matters more to a buyer than the gap itself.
How should models be selected without peeking at the held-out family?
Pick prompts, hyperparameters and checkpoints on a validation family that is neither in training nor in the test fold. Tuning on the held-out family, even informally by looking at a few failures, turns the benchmark into an in-distribution test.
Domain generalization research makes the stakes concrete. DomainBed documents three model selection criteria that differ only in which data is used to choose hyperparameters: a random subset of the training domains, a held-out domain that is neither training nor test, or the test domain itself, which it labels an oracle. Report which one you used, and if any oracle-style selection happened, label the result as an upper bound.
Log every person and system that saw held-out cases. Prompt authors, annotators who wrote training examples, and anyone who debugged the agent should be firewalled from the test family until the run is frozen.
Why is the family, not the case, the unit of uncertainty?
Cases within one family share a trigger, a policy and often the same supplier or customer, so they succeed or fail together. Treating them as independent overstates precision. Evan Miller's paper on adding error bars to evaluations recommends clustered standard errors when questions come in related groups, which is exactly the structure here.
The consequence is that the number of families matters more than the number of cases. A benchmark with thousands of cases in five families supports only five held-out folds and wide intervals on the generalization gap. Report the number of families and cases behind every interval, show per-family results next to any average, and run each case several times so that agent randomness is separated from family difficulty.
Use paired comparisons where you can. Arms B and C run on the same cases, so compare them case by case rather than as two independent averages.
Versioning, release and the conclusions this design will not support
Pin the version of the taxonomy, the task set, the grader and every model, and never compare results across grader versions. tau2-bench's own release notes give a concrete warning: after a grading update fixed task errors in one domain, the maintainers stated that results produced with earlier versions are not comparable with later ones and re-graded affected leaderboard submissions.
Release the task schema, family definitions, grader checks and the reporting plan. Release case content only if the licenses allow benchmark publication, and keep a sealed copy of each held-out family so it can be reused without contamination.
This design will not show that an agent handles every unseen exception, only those like the families tested. It will not separate a family that is genuinely new from one that is merely hard. It will not transfer to companies outside the sampled population unless a company holdout is added. Any result should be described in those terms.
Pre-registration checklist before the first run
At a glance
Pre-registration checklist before the first run
Checklist · 10 items
- Research question and primary metric written down, with the minimum gap that would matter to the decision.
- Population, companies, systems and collection dates recorded, with inclusion and exclusion rules.
- Rights confirmed for benchmark use and for any publication of cases or results.
- Family taxonomy versioned, expert agreement measured, nearest siblings listed.
- Folds, validation family and per-arm case counts fixed.
- Policy text for each held-out family frozen for arms B and C.
- Baselines, grader checks and adjudication rules versioned.
- Uncertainty method, number of trials and clustering unit stated.
- Firewall list of people and systems that may not see held-out cases.
- Statement of conclusions the design will not support, published with any result.
0 of 10 ticked. Tick each item as you go.
Sourcing rights-cleared exception cases for a family holdout
The hard part is rarely the split. It is finding enough closed exception cases, from several companies, with the raise-time record and the resolution intact and permission to use them for evaluation. A family holdout needs depth in many families at once, which a single company's history seldom provides.
SourceX keeps no catalog of ready-made benchmarks; it searches for U.S. businesses whose closed exception cases fit a particular request. The request form is where a benchmark team states scope, volume, language, format, timing and licensing terms, including benchmark use and any publication, and SourceX then coordinates rights review, contracting, preparation and delivery with the licensing businesses.
Frequently asked questions
How many exception families does a holdout benchmark need?
Enough that each held-out fold is one of many, because the family is the unit of uncertainty. With only a handful of families, intervals on the generalization gap will be wide whatever the case count. Fix the number in the pre-registration and report it next to every result.
Should the agent see the written policy for the held-out family?
Run it both ways. With the policy, the test measures whether the agent can apply a rule it has read but never practiced; without it, the test measures inference from general knowledge. The difference between the two is often the most useful number for a deployment decision.
Is holding out whole companies enough?
No, it tests a different thing. A company holdout checks whether the agent transfers to new policies and systems for familiar exception types. A family holdout checks whether it copes with a type of exception it has never seen. Many benchmarks need both, reported separately.
What if experts disagree about which family a case belongs to?
Measure agreement before splitting and adjudicate disagreements with a written rule. Cases that remain ambiguous should go into neither the training side nor the held-out family, because a case that straddles two families leaks one into the other.
Are any accuracy or gap figures published with this protocol?
None. It is a protocol for designing and reporting the benchmark. No model has been run against it here, and no accuracy, gap or interval figures are claimed.
Sources
- scikit-learn's LeaveOneGroupOut builds each cross-validation split by holding out the samples that belong to one group. Source
- SCAN's add-primitive split trains on commands without jump plus the bare jump command and tests on compositional jump commands; its few-shot setting adds increasing numbers of examples with several random samples per size. Source
- DomainBed documents three model selection criteria: a subset of training domains, a held-out domain that is neither training nor test, and an oracle subset of the test domain. Source
- tau2-bench evaluates customer service agents across domains including airline, retail and telecom, and states that results from before its v1.0.1 grading update are not comparable with later results. Source
- WorkArena includes atomic knowledge work tasks, and WorkArena++ composes those atomic components into longer real-world use cases. Source
- Adding Error Bars to Evals recommends clustered standard errors when evaluation questions come in related groups. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.