AI uses for records
Supply chain exception holdout: how to test agents on unseen families
By SourceX Editorial · Updated
10 sections
Short answer
To measure whether a supply chain agent handles exception families it has never seen, group past exceptions into families by shared constraint and approval path, hold whole families out of training, tuning and prompts, and compare scores on held-out families with scores on fresh cases from seen families. This is a study protocol: it gives the design, metrics, uncertainty and a pre-registration checklist, and reports no results.
Evaluation split
Supply chain dataset
Key takeaways
- A random split of exception cases mixes seen and unseen situations and overstates generalization.
- Define families by the constraint the agent must respect and who can authorize an exception, not by reason code alone.
- Score whether every constraint was respected on the way to the final state, not only the final state.
- Treat families as clusters when computing uncertainty, and hold out several families rather than one.
- Pre-register the design, and keep held-out families away from prompts, retrieval indexes and tuning runs.
How do you evaluate supply chain agents on unseen exception families?
Group historical supply chain exceptions into families that share a constraint and an authority path, hold entire families out of everything the agent builder touches, and compare the agent's score on those held-out families with its score on fresh cases from families it did see. The gap between the two is the measure of generalization; the score on seen families alone measures familiarity.
This page is a protocol for running that measurement on procurement and supply chain exception records. It sets out the research question, population, family definitions, sampling, comparison, metrics, uncertainty, leakage controls and a pre-registration checklist. It reports no results: no agent has been scored here, and every count in the page is a placeholder for a team to fill in from its own records.
Why does a random split of exception cases overstate generalization?
A random split puts cases from the same family on both sides, so the test mostly checks whether the agent recognizes situations it has effectively already seen. scikit-learn's documentation states the underlying problem: the independence assumption breaks when data comes in groups of dependent samples, and to know whether a model generalizes to unseen groups, every sample in the validation fold must come from groups absent from the paired training fold. Its LeaveOneGroupOut splitter holds out one group at a time for exactly this reason.
The size of the effect is an empirical question for each dataset, but the direction is well documented. The WILDS benchmark, built from ten datasets with natural shifts across hospitals, cameras, countries, time periods and other domains, found that standard training gave substantially lower out-of-distribution than in-distribution performance on every one. An exception family an agent has never seen is a shift of the same kind.
What counts as an exception family in procurement records?
A family is a set of exceptions that share the constraint the agent must respect and the people who can authorize a way around it. Reason codes are a starting point, but two codes that route to the same approver under the same rule belong in one family, and one code that covers two different authority paths should be split.
Enterprise systems already encode many of these constraints. In Dynamics 365 Supply Chain Management, purchase orders under change management move through draft, in review, rejected, approved, confirmed and finalized statuses; an approved order can be changed only through a request change action, and a finalized order is financially closed and cannot be changed. Its invoice matching compares invoice prices to the purchase order and, under three-way matching, invoice quantities to product receipts, flagging discrepancies beyond set tolerances. Its engineering change process reviews and approves change requests and assesses their impact on existing transactions.
| Family | Trigger in the records | Constraint the agent must respect | Authority path | Evidence that closes it |
|---|---|---|---|---|
| Price variance over tolerance | Invoice price differs from the purchase order beyond the tolerance | Matching policy and tolerance in force on the invoice date | Accounts payable, then buyer or approver | Corrected invoice, approved variance or credit note |
| Quantity short against receipt | Invoice quantity exceeds the received quantity under three-way matching | Pay only for what was received | Receiving, then accounts payable | Additional receipt, partial payment or vendor credit |
| Change after approval | Request to alter an approved or confirmed order | Changes go back through approval; finalized orders cannot change | Requester, approval workflow | Re-approved revision or a new order |
| Specification revision | Engineering change affecting an item on open orders | Revised specification and its effective date | Engineering change approvers | Order updated or canceled against the new revision |
| Supplier not approved | Order or substitution with a supplier outside the approved list | Supplier approval status on the order date | Supplier quality or category manager | Approval granted, or order moved to an approved supplier |
| Late delivery against a customer commitment | Confirmed date later than the date the downstream order needs | Customer delivery constraint and any penalty terms | Planner, then sales or customer service | Expedite, split shipment, reschedule agreed with customer |
| Substitute item | Ordered item unavailable; alternative proposed | Substitution rules and customer or specification approval | Buyer with engineering or customer sign-off | Substitute received and accepted |
The task family holdout and generalization study plan
Fill in every row before any agent is run, and keep the completed plan with the results. The plan is written for one agent and one organization's records; a comparison of several agents uses the same plan with the agent as an extra factor.
At a glance
The task family holdout and generalization study plan
| Design element | Specification | Record to keep |
|---|---|---|
| Research question | Does the agent resolve exceptions from families absent from its training, tuning and prompts as well as fresh cases from families it saw? | Question as written before data access |
| Population | Closed exception cases in a stated date range, entities and systems, with a settled outcome | Date range, entities, systems, inclusion and exclusion rules |
| Unit of analysis | One exception case, scored as one task; families are the unit of holdout | Case and family identifiers |
| Family assignment | Written rule mapping each case to one family by constraint and authority path | Rule text, second-coder agreement on a sample, unassigned cases |
| Holdout selection | Families chosen for holdout by a rule fixed in advance, such as random draw among families above a size floor | Seed or draw record, list of held-out families |
| Comparison | Score on held-out families against score on a held-back sample of cases from seen families | Both case lists, frozen before scoring |
| Baselines | A rules-only or retrieval-only agent, and the human resolution recorded in the case | Baseline definitions and versions |
| Primary metric | Share of tasks where the agent's final system state matches the recorded correct state and every constraint was respected | Scoring rubric and grader instructions |
| Secondary metrics | Constraint violations, escalations to the correct authority, consistency across repeated runs | Per-task logs |
| Uncertainty | Intervals that treat the family as the cluster, plus per-family results | Analysis script and its version |
Which constraint makes the cheapest or fastest choice invalid?
Build the task set so that the obvious answer is wrong in a known share of cases, because that is where an agent's judgment shows. In procurement the obvious choice is usually the cheapest supplier, the fastest delivery or paying the invoice as submitted; the constraint that invalidates it is usually an approval status, a specification revision, a matching rule or a customer commitment.
Score the constraint, not only the end state. tau-bench grades agents by comparing the final database state with an annotated goal state, which is a strong check that the right thing happened. For procurement tasks, add a check that the path was valid: an agent that reaches the right purchase order state by changing a finalized order, or by skipping an approval, should fail the task even though the final state looks correct.
Record each task's invalidating constraint as a field. That lets the analysis report results separately for tasks where the obvious choice was right and tasks where it was wrong, which is often the most informative split in the study.
How many families, runs and baselines does the comparison need?
Treat families, not cases, as the thing you are sampling, because cases within a family are correlated. Evan Miller's paper on adding error bars to evaluations recommends clustered standard errors when questions are drawn in related groups, and inference on question-level paired differences when two models are compared on the same questions. Both apply directly here: cluster by family, and compare agents task by task.
Hold out several families rather than one. A single held-out family produces a single data point about generalization, whatever the number of cases inside it. Set a minimum number of cases per family and a minimum number of held-out families before the study starts, and report the result as inconclusive if the records cannot meet them.
Run each task more than once when the agent is not deterministic. tau-bench introduced pass^k, the chance that all k independent trials of a task succeed, as a reliability measure for agents that act on behalf of users; it fits procurement tasks where one wrong action out of several runs is a real cost.
Report the human resolution recorded in each case as the reference, and say plainly where it was itself contested or reversed. A baseline that is sometimes wrong should not be scored as perfect.
Leakage paths that put a held-out family back into training
Holding out a family fails quietly when any part of the build process sees it. The most common leak is tuning: Gulrajani and Lopez-Paz argued that domain generalization methods without a stated model selection strategy are incomplete, because choosing models or prompts using the held-out domain inflates the result. Choose prompts, tools and thresholds only on seen families.
- Prompt examples or few-shot cases drawn from a held-out family.
- Retrieval indexes built over the full case archive, including held-out cases.
- Policy documents or playbooks that quote held-out cases as examples.
- The same purchase order, supplier dispute or engineering change split across families.
- Tuning runs scored on the held-out families, even once.
- Grader instructions written after looking at held-out cases.
Pre-registration checklist before anyone sees a score
Write and date these items before the first agent run. Any later change goes into a dated amendment that says what changed and why.
At a glance
Pre-registration checklist before anyone sees a score
Checklist · 10 items
- Research question and the comparison that answers it.
- Population boundaries: date range, entities, systems and outcome-settled rule.
- Family assignment rule, second-coder agreement check and handling of unassigned cases.
- Holdout rule, the number of held-out families and the minimum cases per family.
- Primary metric, scoring rubric and the constraint-adherence check.
- Baselines and how the recorded human resolution is treated when it was reversed.
- Number of repeated runs per task and the reliability measure.
- Uncertainty method, with families as clusters.
- Leakage controls and who holds access to held-out cases.
- Conditions under which the result is reported as inconclusive.
0 of 10 ticked. Tick each item as you go.
What this protocol cannot tell you
A held-out family result says how an agent handled the families you held out, in your records, over your period. It does not show how the agent will handle families that never occurred in your history, and it does not transfer to another company with different approval rules without a new study.
The protocol also inherits the quality of the records. If outcomes were recorded inconsistently, or approvals were made by email and never logged, the scoring will be noisy in ways the uncertainty interval does not capture. Record those limits in the report next to the result, once there is one.
Preparing exception families for a study, or for licensing
Exception records with settled outcomes, documented constraints and named authority paths are what make this study possible, and they are also what AI developers look for in licensed operational data. Distributors and manufacturers holding such procurement histories can license selected records through SourceX, which runs rights review, preparation, contracting and delivery.
Large purchasing archives do not need to move to SourceX: they can stay in the company's own storage or ship on encrypted drives. Before any exception record is delivered, the company approves the scope and terms and signs the agreement.
Frequently asked questions
What should the task family holdout and generalization study plan include?
The research question, population, unit of analysis, family assignment rule, holdout selection rule, comparison, baselines, primary and secondary metrics, and an uncertainty method that treats families as clusters. Each element is written and dated before any agent is run.
Can existing reason codes serve as exception families?
They are a starting point. Merge codes that share the same constraint and approver, split codes that cover different authority paths, and have a second person apply the rule to a sample to check that it can be repeated.
How many exception families should be held out?
More than one, because each held-out family is one observation of generalization. The right number depends on how many families your records hold and how many cases each has; fix both minimums in advance and report the result as inconclusive if they are not met.
Does this page report how well supply chain agents generalize?
No. It is a method for measuring generalization on your own records. No agent has been tested for this page, and any counts a team uses come from its own data.
How is this different from testing an agent on our own exceptions before buying?
A pre-purchase test usually samples recent exceptions at random, which mixes seen and unseen situations. This protocol holds out whole families so the score separates familiarity from generalization.
Sources
- scikit-learn notes that the independence assumption breaks for groups of dependent samples, that testing generalization to unseen groups requires validation groups absent from training, and that LeaveOneGroupOut holds out one group per split. Source
- WILDS is a benchmark of ten datasets with natural distribution shifts, and standard training gave substantially lower out-of-distribution than in-distribution performance on each. Source
- Gulrajani and Lopez-Paz argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Source
- Miller recommends clustered standard errors when evaluation questions come in related groups and inference on question-level paired differences when comparing two models. Source
- tau-bench scores agents by comparing the final database state with an annotated goal state and introduces pass^k, the chance that all k trials of a task succeed. Source
- Dynamics 365 purchase orders under change management move through draft, in review, rejected, approved, confirmed and finalized; approved orders change only through request change, and finalized orders cannot be changed. Source
- Dynamics 365 three-way matching compares invoice prices with the purchase order and invoice quantities with product receipts, and flags discrepancies that exceed tolerances. Source
- Dynamics 365 engineering change management reviews and approves change requests and orders and assesses their impact on existing transactions. Source
See if your company qualifies
A short company assessment. No data uploads are needed.