Skip to content

AI uses for records

Business exception generalization benchmark: a holdout protocol

By SourceX Editorial · Updated

10 sections

Short answer

This is a protocol, not a result. To build a business exception generalization benchmark, group resolved exceptions into families by trigger and policy, hold out every case of one family from training, prompts and model selection, and test on it with and without its written policy alongside unseen cases from seen families. Rotate the held-out family and treat families as the unit of uncertainty. No results are reported.

Evaluation split

Business exception

This page covers business exception generalization benchmark: a holdout protocol.Illustrative

Key takeaways

  • A random split lets an agent recognize familiar exception families, so it cannot show generalization to a new type.
  • Define families from the raise-time record by trigger and policy, and list each family's nearest sibling.
  • Compare seen-family cases, the held-out family with its policy, and the held-out family without it.
  • Select models on a separate validation family; any tuning on the test family makes the result an upper bound.
  • Treat the family as the unit of uncertainty and report per-family results, not only a pooled score.

How do you hold out an entire exception type to test agent generalization?

Group resolved business exceptions into families that share a trigger and a resolution policy, then remove every case of one family from anything the agent or its developers see: training data, prompts, examples and model selection. Test on that family alongside unseen cases from the families that were seen, rotate which family is held out, and treat the family, not the case, as the unit of uncertainty.

This page is a protocol. It sets out the question, population, split, variables, comparisons, metrics, uncertainty handling and limits for a business exception generalization benchmark, and it reports no results. Any numbers below are design parameters or labelled illustrations.

Why can't a random split show that an agent generalizes?

Because a random split puts cases from every exception family on both sides. An agent can then score well by recognizing the family and replaying how similar cases were resolved, which measures recall of familiar patterns, not the ability to handle an exception it has never met. In production, the costly failures come from the new kind of exception: a supplier sends a credit memo in a format nobody has seen, or a policy creates a new category of refund.

Grouped splitting is the standard remedy. scikit-learn's LeaveOneGroupOut, for instance, builds each split by holding out all samples that belong to one group, given an array that assigns every sample to a group. Here the group is the exception family.

The closest research precedent is compositional generalization. The SCAN benchmark's add-primitive split trains on every command that does not use jump, plus the bare jump command alone, and tests on all compositional commands that use jump. That is the shape this protocol borrows: the agent may know that an exception family exists, through its written policy, without having seen a single worked case of it.

Existing agent benchmarks are organized in ways that make this possible but do not usually do it. tau2-bench groups tasks by domain, such as airline, retail and telecom, which is a coarser holdout than an exception family. WorkArena distinguishes atomic tasks from WorkArena++ tasks that compose those atomic components into longer use cases, which tests composition rather than an unseen exception class.

What counts as an exception family?

A family is a set of cases that share a trigger condition and a governing resolution policy. Define families from the record as it stood when the exception was raised, not from how it was resolved, otherwise the family label leaks the answer.

Granularity decides what the benchmark measures. Families that are too broad, such as all invoice problems, leave nothing new in the held-out set. Families that are too narrow, such as price variance on one product line, leave a near-identical sibling in training. Record each family's nearest sibling and check how much of the held-out family's resolution logic the sibling already teaches. The table shows illustrative families from accounts payable and order operations.

What counts as an exception family?
Illustrative familyTriggerGoverning policyNearest sibling to watch
Price varianceInvoice unit price differs from the PO price beyond tolerancePrice tolerance and approval policyFreight or surcharge variance
Quantity mismatchInvoiced quantity exceeds received quantityReceipt matching policyMissing receipt
Duplicate invoiceSame supplier, amount and reference seen beforeDuplicate payment controlCredit memo applied twice
Missing receiptInvoice arrives before any goods receiptHold and follow-up policyQuantity mismatch
Address change after shipmentCustomer changes destination once the order has shippedCarrier intercept and fee policyAddress change before shipment
Substitution requestOrdered item unavailable and a substitute is proposedSubstitution and customer consent rulesPartial shipment and backorder

The task family split and benchmark reporting plan

This is the plan to register before any model run. Each row names a design component, the choice this protocol makes, the reason, and what the report must state about it.

At a glance

The task family split and benchmark reporting plan

Sampling follows the families, not the raw case stream. Draw cases within each family and company with a fixed random seed, cap any single company's share of a family so one firm's habits do not define it, and keep cases from the same underlying incident, such as several invoices from one disputed shipment, on the same side of every split. Record for each case the variables a reviewer will need to explain a failure: company, system, family, raise date, resolution, whether the resolution was escalated, whether more than one resolution was acceptable, and the policy version in force on the raise date.

Define the population window so that every held-out case is closed and its outcome settled. An exception resolved last week may still be reversed, and a reversed resolution makes the recorded ground truth wrong. Leave a settling period after the last case, state it, and exclude cases reopened inside it.

The task family split and benchmark reporting plan
ComponentDesign choiceReasonWhat the report states
Research questionDoes accuracy on an unseen exception family fall short of accuracy on unseen cases of seen families, and by how much?Separates new-family generalization from case-level generalizationThe question verbatim and the primary metric
PopulationClosed, rights-cleared exception cases from several companies, each with its record at raise time and the recorded resolutionGround truth must exist and be licensable for benchmark useCompanies, systems, collection dates, inclusion and exclusion rules
Family assignmentTwo domain experts assign families from the raise-time record; disagreements adjudicated before splittingUnstable labels make the holdout meaninglessTaxonomy version, agreement rate, adjudication rule
SplitLeave one family out, rotated over every family with enough cases; a separate validation family is held out from both training and testEvery family gets a turn and model selection never touches the test familyFamilies held out per fold and case counts per arm
Policy exposureHeld-out family tested with and without its written policy documentDistinguishes reading a new rule from inferring itExact policy text supplied in each condition
BaselinesRoute-to-human rule, most frequent resolution, and the same agent with no tool accessShows whether the agent beats trivial strategies on the new familyEach baseline's definition and version
GraderEnd-state checks against the recorded resolution, with expert adjudication where several resolutions are validBusiness exceptions often allow more than one correct outcomeGrader version, check list and adjudication rate
MetricsResolution accuracy, correct escalation rate, forbidden action rate, reported per familyA wrong irreversible action is worse than a needless escalationPer-family values, never only a pooled score
UncertaintyClustered intervals by family, repeated trials per caseCases within a family are not independentNumber of families, cases and trials behind every interval

Which comparisons does the design isolate?

Three arms per fold, each answering a separate question. Running only the held-out arm tells you a number with nothing to compare it to.

  • Arm A, seen families, unseen cases: the in-distribution reference. Cases come from families present in training, but the cases themselves were never seen.
  • Arm B, held-out family with its policy: the agent receives the written policy for the new family but no worked cases. The gap between A and B is the new-family generalization gap.
  • Arm C, held-out family without its policy: the same cases with no policy text. The gap between B and C shows how much the written rule helps compared with inference from general knowledge.
  • Optional dose arm: add one, two, four and then more worked cases from the held-out family, in the spirit of SCAN's few-shot conditions, which add increasing numbers of examples with several random samples at each size. This shows how quickly the gap closes, which matters more to a buyer than the gap itself.

How should models be selected without peeking at the held-out family?

Pick prompts, hyperparameters and checkpoints on a validation family that is neither in training nor in the test fold. Tuning on the held-out family, even informally by looking at a few failures, turns the benchmark into an in-distribution test.

Domain generalization research makes the stakes concrete. DomainBed documents three model selection criteria that differ only in which data is used to choose hyperparameters: a random subset of the training domains, a held-out domain that is neither training nor test, or the test domain itself, which it labels an oracle. Report which one you used, and if any oracle-style selection happened, label the result as an upper bound.

Log every person and system that saw held-out cases. Prompt authors, annotators who wrote training examples, and anyone who debugged the agent should be firewalled from the test family until the run is frozen.

Why is the family, not the case, the unit of uncertainty?

Cases within one family share a trigger, a policy and often the same supplier or customer, so they succeed or fail together. Treating them as independent overstates precision. Evan Miller's paper on adding error bars to evaluations recommends clustered standard errors when questions come in related groups, which is exactly the structure here.

The consequence is that the number of families matters more than the number of cases. A benchmark with thousands of cases in five families supports only five held-out folds and wide intervals on the generalization gap. Report the number of families and cases behind every interval, show per-family results next to any average, and run each case several times so that agent randomness is separated from family difficulty.

Use paired comparisons where you can. Arms B and C run on the same cases, so compare them case by case rather than as two independent averages.

Versioning, release and the conclusions this design will not support

Pin the version of the taxonomy, the task set, the grader and every model, and never compare results across grader versions. tau2-bench's own release notes give a concrete warning: after a grading update fixed task errors in one domain, the maintainers stated that results produced with earlier versions are not comparable with later ones and re-graded affected leaderboard submissions.

Release the task schema, family definitions, grader checks and the reporting plan. Release case content only if the licenses allow benchmark publication, and keep a sealed copy of each held-out family so it can be reused without contamination.

This design will not show that an agent handles every unseen exception, only those like the families tested. It will not separate a family that is genuinely new from one that is merely hard. It will not transfer to companies outside the sampled population unless a company holdout is added. Any result should be described in those terms.

Pre-registration checklist before the first run

At a glance

Pre-registration checklist before the first run

Checklist · 10 items

  1. Research question and primary metric written down, with the minimum gap that would matter to the decision.
  2. Population, companies, systems and collection dates recorded, with inclusion and exclusion rules.
  3. Rights confirmed for benchmark use and for any publication of cases or results.
  4. Family taxonomy versioned, expert agreement measured, nearest siblings listed.
  5. Folds, validation family and per-arm case counts fixed.
  6. Policy text for each held-out family frozen for arms B and C.
  7. Baselines, grader checks and adjudication rules versioned.
  8. Uncertainty method, number of trials and clustering unit stated.
  9. Firewall list of people and systems that may not see held-out cases.
  10. Statement of conclusions the design will not support, published with any result.

0 of 10 ticked. Tick each item as you go.

Sourcing rights-cleared exception cases for a family holdout

The hard part is rarely the split. It is finding enough closed exception cases, from several companies, with the raise-time record and the resolution intact and permission to use them for evaluation. A family holdout needs depth in many families at once, which a single company's history seldom provides.

SourceX keeps no catalog of ready-made benchmarks; it searches for U.S. businesses whose closed exception cases fit a particular request. The request form is where a benchmark team states scope, volume, language, format, timing and licensing terms, including benchmark use and any publication, and SourceX then coordinates rights review, contracting, preparation and delivery with the licensing businesses.

Frequently asked questions

How many exception families does a holdout benchmark need?

Enough that each held-out fold is one of many, because the family is the unit of uncertainty. With only a handful of families, intervals on the generalization gap will be wide whatever the case count. Fix the number in the pre-registration and report it next to every result.

Should the agent see the written policy for the held-out family?

Run it both ways. With the policy, the test measures whether the agent can apply a rule it has read but never practiced; without it, the test measures inference from general knowledge. The difference between the two is often the most useful number for a deployment decision.

Is holding out whole companies enough?

No, it tests a different thing. A company holdout checks whether the agent transfers to new policies and systems for familiar exception types. A family holdout checks whether it copes with a type of exception it has never seen. Many benchmarks need both, reported separately.

What if experts disagree about which family a case belongs to?

Measure agreement before splitting and adjudicate disagreements with a written rule. Cases that remain ambiguous should go into neither the training side nor the held-out family, because a case that straddles two families leaks one into the other.

Are any accuracy or gap figures published with this protocol?

None. It is a protocol for designing and reporting the benchmark. No model has been run against it here, and no accuracy, gap or interval figures are claimed.

Sources

  • scikit-learn's LeaveOneGroupOut builds each cross-validation split by holding out the samples that belong to one group. Source
  • SCAN's add-primitive split trains on commands without jump plus the bare jump command and tests on compositional jump commands; its few-shot setting adds increasing numbers of examples with several random samples per size. Source
  • DomainBed documents three model selection criteria: a subset of training domains, a held-out domain that is neither training nor test, and an oracle subset of the test domain. Source
  • tau2-bench evaluates customer service agents across domains including airline, retail and telecom, and states that results from before its v1.0.1 grading update are not comparable with later results. Source
  • WorkArena includes atomic knowledge work tasks, and WorkArena++ composes those atomic components into longer real-world use cases. Source
  • Adding Error Bars to Evals recommends clustered standard errors when evaluation questions come in related groups. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies