AI uses for records
Retail promotion evaluation holdout: a protocol for unseen promotions
By SourceX Editorial · Updated
10 sections
Short answer
This is a protocol, not a study, and it reports no results. To evaluate retail agents on previously unseen promotions, group tasks into promotion families by discount target, combination rule, precedence, trigger and eligibility. Hold out whole families before any run, compare accuracy on seen and unseen families, and report the gap with a family-level interval.
Evaluation split
Retail promotion evaluation
Key takeaways
- Split evaluation tasks by promotion family, never at random, or near-copies of test tasks leak into the agent's examples.
- Define families from platform settings that change the answer: target, combination rule, precedence, trigger and eligibility.
- Take gold answers from the platform's recorded calculation at order time and set manual overrides aside as escalation tasks.
- The family is the unit of analysis, so intervals come from resampling families and small held-out sets are pilots.
- This page publishes the method only; it reports no measured results.
How do you evaluate retail agents on previously unseen promotions?
Hold out whole promotion families, not individual tasks. Group every evaluation task by the promotion mechanics it depends on, assign some families entirely to the test set before any run, and compare the agent's accuracy on families it has seen in its instructions or examples with families it has not. The gap between the two, reported with an interval at the family level, is the measure of generalization.
What follows is a measurement plan rather than a finished study. It sets out the question, population, sampling, variables, comparison conditions, metrics, uncertainty, limits and a pre-registration checklist for a policy family holdout and generalization experiment. No results are reported here, and every count in an example is labelled illustrative.
Why a random task split overstates how well an agent handles new promotions
Tasks built on the same promotion share its rules, so a random split puts near-copies of each test task into the examples or instructions the agent was tuned on. The score then measures recall of familiar rules rather than the ability to apply a new one.
The scikit-learn documentation states the general problem well: when samples come in dependent groups, the independence assumption breaks, and to learn whether a model generalizes to unseen groups every validation sample must come from groups absent from the paired training fold. Its group splitters, such as GroupKFold and LeaveOneGroupOut, exist for exactly this case. The WILDS benchmark makes the same distinction at a larger scale, defining domain generalization as performance on test domains disjoint from the training domains.
Public agent benchmarks do not yet cover this ground. The retail policy in the original tau-bench repository covers authentication, cancelling and modifying pending orders, returns and exchanges, with no promotion or discount rules at all, and its maintainers now mark those tasks as outdated in favor of a newer repository. A promotion holdout therefore needs its own task set built from real promotion configurations and orders.
What makes two promotions the same family?
A family is a set of promotions that an agent could handle with the same reasoning. Define it from the configuration options of the commerce platform the promotions ran on, because those options are what change the valid answer.
Platform documentation shows how much those options differ. Shopify's help center separates product, order and shipping discounts, applies them in that order and lets merchants set which classes combine. Adobe Commerce cart price rules carry a priority, where a lower number applies first, and a setting that discards lower-priority rules once a rule applies. WooCommerce coupons have an individual use only setting that blocks combination with other coupons. Two promotions with the same percentage but different combination settings belong in different families.
| Family axis | Example values | Platform anchor | Why it changes the answer |
|---|---|---|---|
| Discount target | Line item, order subtotal, shipping | Shopify product, order and shipping discount classes | Changes the base the discount applies to |
| Combination rule | Stacks with all, stacks within class, exclusive | Shopify combinations; WooCommerce individual use only | Decides which promotions can coexist on one cart |
| Precedence | Priority order, stop further rules | Adobe Commerce priority and discard subsequent rules | Decides which promotion wins a conflict |
| Trigger | Code entered, automatic, threshold, buy X get Y | Platform rule conditions | Decides whether the promotion applies at all |
| Eligibility | Customer segment, product collection, channel, date window | Platform rule conditions | Decides which customers and items qualify |
| After-sale behavior | Refund allocation on return, re-evaluation on exchange | Merchant return policy | Decides the correct refund or price after a change |
The policy family holdout experiment, element by element
This is the asset: the design a team fills in and freezes before collecting any agent output. Each element states what to specify and the rule that keeps the experiment honest.
At a glance
The policy family holdout experiment, element by element
| Element | What to specify | Rule that keeps it honest |
|---|---|---|
| Research question | Does accuracy on promotion tasks drop when the promotion family was absent from the agent's instructions and examples? | One primary question; secondary questions labelled exploratory |
| Population | Closed orders with at least one promotion evaluated, from named stores and a stated date range | Report stores, platforms and dates; no pooling across platforms without stating it |
| Unit of analysis | A task: cart snapshot, promotion versions live at that instant, customer request, expected action | One task per order event; repeated orders from one customer flagged |
| Family assignment | Each task tagged with its family from the axis table | Tags fixed by two reviewers before splitting; disagreements resolved and logged |
| Holdout | Whole families assigned to test by a recorded random seed, plus one held-out combination of seen families | No task from a held-out family in prompts, examples or tuning |
| Comparison conditions | Seen family, unseen family, unseen combination of seen families | Same agent, same tools, same instructions apart from family content |
| Outcome variables | Valid promotion set, application order, per-line adjustment, cited rule, escalation | Each scored separately; no single blended score as primary |
| Covariates | Cart size, number of promotions offered, customer segment, channel | Reported per condition so imbalance is visible |
| Uncertainty | Family-level cluster bootstrap for the gap; per-family proportion intervals | Resample families, not tasks |
| Stopping rule | Fixed number of runs per task decided in advance | No added runs after looking at results |
Where gold answers for held-out promotions come from
Take the gold answer from the platform's own calculation at order time, checked against the promotion configuration version that was live, and never from a reviewer's reading of the promotion text. The platform record shows what the rules actually did; a reviewer shows what they think the rules say.
Exclude orders where a staff member overrode a discount, applied a manual adjustment or issued a goodwill credit, because the recorded outcome no longer reflects the rules. Keep a sample of those orders aside as escalation tasks, where the correct agent answer is to hand off rather than compute.
Where the platform's calculation is itself suspect, for example a version affected by a known defect in rule handling, either drop the affected window or label those tasks as disputed and report them separately. Adobe has published a quality patch for cart rules whose discard subsequent rules setting did not work as intended, which is a reminder that the engine's output is evidence, not truth, and needs its version recorded.
Every task record should also store the promotion configuration as data the agent can be given or denied, separately from the free-text description a merchant wrote for shoppers. The marketing text often omits combination settings, exclusions and priority, so an agent that reads only the banner copy is being tested on guesswork. Keeping both lets the study measure how much each one helps.
Have a second reviewer re-derive a random tenth of the gold answers from the configuration and order record without seeing the stored answer. Disagreements point to tagging errors, missing configuration versions or orders that should have been excluded, and the agreement rate belongs in the write-up next to the results.
Cart, stock and customer changes that move the right answer after checkout
Promotion tasks are rarely a single calculation. The valid action often changes because something linked to the cart changed between checkout and the customer's request, and the task record must carry that state.
At a glance
Cart, stock and customer changes that move the right answer after checkout
- Inventory: the free item in a buy X get Y offer goes out of stock before fulfillment, so the agent must apply the merchant's substitution or adjustment rule.
- Returns: a customer returns one item from a discounted bundle, and the refund depends on how the order discount was allocated across lines.
- Exchanges: swapping a size or color re-evaluates eligibility if the new variant sits outside the promoted collection.
- Time: the promotion window closes between cart creation and checkout, so the version live at the order instant governs.
- Customer state: a loyalty tier or segment changes, which changes eligibility for segment-limited promotions.
- Shipping: an address change moves the order out of a free shipping zone or threshold, and the platform's shipping discount precedence decides what remains.
How many families, which baselines and what interval a holdout result needs
The family, not the task, is the independent unit, so the design needs enough families on each side of the split to say anything. A study with dozens of tasks but two held-out families has a sample size of two for the question that matters.
Use three baselines. A rule replay, re-running the platform's own engine on the snapshot, sets the ceiling and catches broken gold labels. The same agent given the full promotion configuration as structured data separates rule reading from rule finding. A seen family control shows the agent's accuracy when nothing is new. Report the gap between seen and unseen families with a family-level bootstrap interval, and per-family accuracy with a binomial interval such as the Wilson interval, which NIST's Dataplot reference uses as its default for proportions.
Agents are not deterministic, so run each task more than once. The tau-bench paper's pass^k metric, the chance that all k independent trials of a task succeed, averaged across tasks, measures consistency; reporting pass^1 and one higher k for each condition shows whether unseen families make the agent unreliable as well as less accurate.
An illustrative scale for a first study, not a recommendation from data: 24 families split 16 seen and 8 unseen, 15 tasks per family and 4 runs per task. Treat any plan with fewer than 6 unseen families as a pilot.
Pre-registration checklist before the first run
Record each item, date it and store it where the team cannot edit it after the first run. Anything decided after seeing results is reported as exploratory.
At a glance
Pre-registration checklist before the first run
Checklist · 9 items
- Primary question and the single primary metric.
- Population: stores, platforms, date range and exclusion rules, including manual overrides.
- Family axes, tagging guide and the two reviewers who tagged.
- Random seed and the list of held-out families and held-out combinations.
- Agent version, model version, instructions, tools and the examples shown, with proof that no held-out family appears in them.
- Runs per task, the k values for consistency and the stopping rule.
- Interval methods for the gap and for per-family accuracy.
- How disputed gold labels and escalation tasks are scored and reported.
- What result would count as no meaningful gap, stated before running.
0 of 9 ticked. Tick each item as you go.
What the protocol cannot tell you
A holdout study measures generalization within the families and platforms you sampled. It does not show that an agent will handle a promotion mechanic no store in the sample ever ran, and it cannot separate the agent's reasoning from what its underlying model may have seen during pretraining about public platform documentation.
It also says nothing about customer outcomes such as satisfaction or repeat purchase, about fraud and coupon abuse, or about promotions on platforms with different calculation order. Results from one platform should not be read as results for another, since precedence and combination rules differ. Rights and privacy review of the order data, including customer identifiers in carts and messages, belongs before collection, with a retail operations expert confirming the family tags.
Finally, a small gap is not proof of robustness. If the held-out families happen to be simpler than the seen ones, the comparison flatters the agent, which is why covariates such as cart size and the number of promotions offered are reported per condition and why family difficulty should be judged by the rule replay and reviewers before any agent runs.
Sourcing promotion histories for a holdout study
Real promotion histories, with configuration versions, carts and the orders and returns that followed, sit with retailers and ecommerce operators rather than in public benchmarks. SourceX helps retailers turn selected order and promotion records into licensed datasets for AI, handling qualification, rights review, contracting, preparation and delivery.
A merchant can find out whether its promotion history qualifies through a short company assessment with no data uploads, and its order records are delivered only after it approves the scope and terms and signs the agreement.
Frequently asked questions
What is a policy family holdout?
It is an evaluation split where every task tied to a group of similar promotions goes entirely to the test set. The agent never sees those promotion mechanics in its instructions or examples, so its test score shows how well it applies rules it has not met before.
Why not just add new promotions to an existing benchmark?
Adding a few promotions without grouping them lets near-copies leak between the examples and the test set. The group split is what makes the comparison meaningful, and it needs enough families on each side to support an interval.
Does this page report how well current agents do?
No. It publishes the method only. Any figures on the page are illustrative design choices, and results would need a pre-registered study on real promotion and order data.
How many held-out promotion families are enough?
There is no fixed number, because the family is the unit of analysis and the interval width depends on how much families differ. Plans with only a handful of unseen families should be treated as pilots, and the count should be fixed before any runs.
Can synthetic promotions replace real ones?
Synthetic promotions can fill gaps in the axis grid, but they lack the configuration conflicts, overrides and after-sale changes found in real order histories. If used, report them as a separate condition rather than mixing them into the real families.
Sources
- The scikit-learn user guide says grouped samples break the independence assumption and that testing generalization to unseen groups requires validation groups absent from training, which GroupKFold and LeaveOneGroupOut enforce. Source
- The WILDS benchmark defines domain generalization as generalizing to test domains disjoint from the training domains. Source
- The original tau-bench retail policy covers authentication, cancelling and modifying pending orders, returns and exchanges, without promotion rules; the repository marks its tasks as outdated. Source
- The tau-bench paper defines pass^k as the chance that all k independent trials of a task succeed, averaged across tasks. Source
- Shopify applies product discounts first, order discounts to the reduced subtotal next and shipping discounts last, with combinations set per discount. Source
- Adobe Commerce applies the cart price rule with the lowest priority number first and can discard subsequent rules once a rule applies. Source
- WooCommerce's individual use only setting prevents a coupon from being used with other coupons. Source
- Adobe's quality patch ACSD-49480 addresses cart price rules where Discard Subsequent Rules did not work as intended. Source
- NIST's Dataplot reference uses the Wilson method as its default confidence interval for a proportion. Source
See if your company qualifies
A short company assessment. No data uploads are needed.