Skip to content

AI uses for records

Unseen policy agent evaluation: hold out rules, not just cases

By SourceX Editorial · Updated

9 sections

Short answer

Unseen policy agent evaluation asks whether an agent follows rules handed to it at run time rather than rules it already learned. Hold out whole policies, not only cases, because the number of distinct policies is the sample size; in τ²-bench, train and test tasks share one policy per domain. Include rules that contradict common practice, score prior intrusion, and seal test policies from everyone tuning the agent.

Evaluation split

Unseen policy agent

This page covers unseen policy agent evaluation: hold out rules, not just cases.Illustrative

Key takeaways

  • An unseen policy test holds out the rules themselves; a held-out case set under a practiced policy cannot show adaptation.
  • The number of distinct policies, not cases, sets the sample size: τ²-bench tests 3 policies and SOP-Bench 15 procedures, all public.
  • Grade test policies by distance from what the agent knows, from a known public rule to a counterfactual one that reverses common practice.
  • Prior intrusion, the share of failures that follow the familiar rule, separates not reading from not reasoning.
  • Test policies leak through prompts, examples, retrieval, splits, graders, publication and people; seal them and freeze the agent before unsealing.

What does an unseen policy evaluation test that a held-out case set cannot?

It tests whether an agent can read a rulebook it has never met and act on it, which a held-out case set cannot show when every test case runs under rules the agent already practiced. A held-out case asks whether the agent can serve a new customer under the refund policy it knows. An unseen policy asks whether it can serve any customer under a refund policy handed to it a minute ago. Business deployments need the second skill whenever rules differ by client contract, region or quarter.

Model builders already ship systems designed around supplied rules. OpenAI describes gpt-oss-safeguard as reasoning over a developer-provided policy at inference time, which it says lets developers revise policies iteratively. Meta's Llama Guard paper described adapting its classifier to other taxonomies through zero-shot or few-shot prompting. Any product that promises to follow your policy is making an unseen-policy claim, and only an unseen-policy test can check it.

This is a separate question from following a policy that has since expired, settling two rules that conflict, or keeping test cases out of a training corpus; those failures have their own guides. Here the rule itself is the new element, so the design has to hold rules out, find out whether the agent's errors come from what it already believed, and stop the test rules from leaking into the work that prepares the agent.

Why the policy, not the case, is the sample

Because a score on many cases under one policy says a great deal about that policy and almost nothing about the next one. An evaluation with three policies and two thousand cases supports a claim about adapting to new policies with three observations of policy-level variation, however narrow the case-level interval looks.

Public benchmarks show the arithmetic. They were built to measure how well agents follow the policies or procedures they are given, not to estimate how results vary from one unseen policy to the next, so this is a note on scope rather than a criticism. In τ²-bench, each domain's environment loads one policy file unless the caller passes a different policy string. The airline domain's 50 tasks are split 30 for training and 20 for testing, and retail's 114 tasks 74 and 40; the changelog entry for version 0.2.1, dated November 7, 2025, added these splits with a Gymnasium interface for reinforcement learning. An agent trained on the train split has practiced under the very policy it is tested on. The three policy files were first committed on June 10, 2025 and have not been edited since, so they have been public for more than a year.

SOP-Bench spreads its cases across more procedures. At commit f6d4385 its 14 benchmark folders hold 15 procedure files and 2,154 rows in their test files with expected outputs, which the README rounds to more than 2,000 tasks across 12 industrial domains. That is about 144 cases per procedure. For an unseen-policy claim the count that matters is still 15, and every procedure is public.

The recount, with the files behind each figure:

Why the policy, not the case, is the sample
Benchmark and commitPolicy or procedure filesTask rowsSplitPolicy shared across split?
τ²-bench airline, 267d1611 (policy.md)5030 train, 20 testYes
τ²-bench retail, 267d1611 (policy.md)11474 train, 40 testYes
τ²-bench telecom, 267d1611 policy (main_policy.md joined with tech_support_manual.md)2,285 full; 114 base74 train, 40 test, drawn from the base setYes
SOP-Bench, 14 folders, f6d438515 (one folder adds a second version)2,154Test files onlyNo split; every procedure is public
Two panels compare task rows with policy files: in τ²-bench, 50 airline, 114 retail and 114 telecom base tasks each run under one policy file, three in all, and in SOP-Bench 2,154 test rows run under 15 procedure files, so an unseen-policy claim rests on 3 or 15.
Task rows versus policy files in two public benchmarks, recounted at commits 267d161 (τ²-bench) and f6d4385 (SOP-Bench), 27 October 2026.

Four distances between a test policy and what the agent already knows

Not every unseen policy is equally unseen, so grade each test policy by how far it sits from rules the agent could already hold. A public policy that has circulated for years tests recall as much as reading. A familiar policy with one value changed, such as a return window cut from 30 to 14 days, tests whether the agent reads the number. A new clause tests whether it can apply a rule with no template. A counterfactual rule, one that reverses common practice, tests whether the supplied text outweighs the agent's prior.

The last rung matters most and is the hardest to pass. Wu and colleagues built counterfactual variants of 11 tasks that keep the reasoning steps of a familiar task but change its rules; language models performed nontrivially on the variants, yet performance degraded substantially and consistently against the default versions. A rule-following agent feels the same pull, because the familiar rule is where it falls back.

Label every test policy with its rung before the run:

Four distances between a test policy and what the agent already knows
DistanceExample (illustrative)What a pass showsTypical failure
Known public policyA published baggage rule unchanged for yearsRecall plus application; little about readingRight for the wrong reason, invisible in the score
Edited valueReturn window cut from 30 to 14 daysThe agent reads values instead of assuming themApplies the old number
New clauseA second approver above a credit limit set per clientThe agent applies a rule it has no template forSkips the step entirely
Counterfactual ruleReplacement shipped before the return arrives, reversing a return-first ruleThe supplied text overrides the agent's priorApplies the common-practice rule

Prior intrusion: the error that shows the agent did not read the rule

Prior intrusion is a wrong action that matches a familiar or earlier rule instead of the supplied one, and its share of errors tells you whether failures come from not reading or from not reasoning. Measuring it needs cases where the supplied rule and the likely prior give different correct actions. A case where both rules agree cannot reveal which one the agent used.

Score three numbers for every test policy: success on divergent cases, where the rules disagree; success on convergent cases, where they agree; and the prior intrusion rate, the share of failed divergent cases whose action matches the prior rule. A wide gap between convergent and divergent success with high intrusion means the agent is answering from memory. A similar gap with low intrusion points to reasoning or tool errors instead.

Rule-swap twins make the test sharper. Run one case, with the same customer, history and system state, under two versions of a policy that should lead to different actions, and check that the agent's action changes when, and only when, the rule does. A twin that flips without reason exposes noise; a twin that never flips exposes an agent that is not reading.

How test policies leak into the work that prepares an agent

They leak through every step where someone tunes the agent after seeing results, not only through training data. A test policy is spent once it shapes a prompt, an example, a retrieval index, a grader or a training split. Statistics has a precise version of this warning: Dwork and colleagues showed in Science in 2015 that analyses chosen after seeing results on the same data void the usual statistical assurances, and they proposed a way to reuse a holdout set safely.

At a glance

How test policies leak into the work that prepares an agent

The routes in an agent project are concrete, and each has a control:

  • Prompt iteration: engineers rewrite the system prompt after reading failures on test policies. Control: iterate on development policies only, and return aggregate scores, not transcripts, from test runs.
  • Worked examples: few-shot examples are drawn from cases decided under test policies. Control: build examples from development policies and log where each came from.
  • Retrieval: the agent's document index holds the test policies or cases decided under them. Control: check a hash of the index snapshot before every run.
  • Training splits: a train split shares the test policy, as public benchmark splits often do. Control: split by policy and record the split key.
  • Graders and judges: a judge prompt is tuned on test-policy transcripts. Control: tune on development policies and freeze the grader before unsealing.
  • Publication: a test policy appears on a website, in a paper or in a repository. Control: record the first public date and retire the policy from unseen claims after it.
  • People: the same person writes test policies and tunes the agent. Control: separate authors, with an access log on the sealed pool.

A sealed policy pool, step by step

Seal whole policies, freeze the agent before unsealing them, and spend each one once. Real policy histories fill the pool faster than rules written from scratch: a business that changed its credit terms after a model's training data ends has, for that model, an unseen policy with real cases decided under it (illustrative). Date both the change and the first time the text appeared in public, because an internal policy that was never posted can stay unseen far longer than one on a website.

The order of the steps is what an auditor needs to see:

Checklist · 7 items

  1. Collect a pool of policies with version numbers, effective dates and the date each first became public, if it ever did.
  2. Assign policies, not cases, to development and test, and keep rule-swap twins together on one side.
  3. Label each test policy with its distance rung and its share of divergent cases.
  4. Seal the test pool: restricted access, an access log and a hash of every file.
  5. Freeze the agent: model version, prompt, tools, retrieval snapshot and grader, each recorded with a hash.
  6. Unseal, run every case the planned number of times, and report per-policy results before any average.
  7. Retire spent policies from unseen claims, keep them as a regression set, and replace them with new or newly changed ones.

0 of 7 ticked. Tick each item as you go.

Could a lower score mean the new policies are simply harder?

Yes, and the design has to separate novelty from difficulty before blaming either. New policies may be longer, vaguer or more conditional than familiar ones, and newer cases may come from a shifted mix of customers. A drop on unseen policies then blends three effects: the agent not reading, the rules being harder and the cases being different.

Three controls pull them apart. Pair each test policy with a known policy of similar length and clause count and compare the two. Use rule-swap twins, which hold the case fixed and change only the rule. And have two people who did not write the policy answer a sample of cases from the text alone, then drop or rewrite cases they disagree on, since an ambiguous policy measures its writer rather than the agent. When the matched gap persists and intrusion is high, the agent is the problem.

Do not chain scores across pool versions as if they were one series. A refreshed pool with new policies is a new test: report it as a new baseline and keep retired policies as a regression check, so historical comparisons stay honest.

Time-aware data card and change-impact checklist for a policy pool

Each policy in the pool gets one card entry, and the checklist runs whenever a policy changes, becomes public or is used in a run. The example entries are illustrative.

Checklist · 7 items

  1. Did the policy text change? Issue a new version and hash; never edit a sealed version.
  2. Did the text appear anywhere public? Record the date and retire it from unseen claims.
  3. Did anyone outside the evaluation team read it? Log the access and decide whether to retire it.
  4. Did the model, prompt, retrieval index or grader change since the last run? Rerun the development policies first.
  5. Do new cases come from a different period or customer mix than earlier ones? Report them as a separate slice.
  6. Are results shown per policy and per distance rung, with the number of policies printed beside any average?
  7. Is every divergent case still divergent after the latest policy edit?

0 of 7 ticked. Tick each item as you go.

Time-aware data card and change-impact checklist for a policy pool
Card fieldWhat to recordExample entry (illustrative)
Policy id and versionStable id, version number and a hash of the textCREDIT-7 version 3, hash stored
Effective datesWhen the version took effect and when it was replacedIn force since 2026-03-01
First public dateEarliest date the text appeared outside the organization, or neverNever published
Distance rungKnown, edited value, new clause or counterfactualNew clause
Split and sealDevelopment or test, sealed status and where the access log livesTest; sealed; three named readers
CasesCount, divergent share and agreement between two independent readers on a sample120 cases; 70% divergent; readers agreed on 46 of 50
Exposure logEvery run, prompt, index or grader that touched the textUnsealed once, for the frozen agent
RetirementDate moved to regression use, and whyNot yet retired

Sourcing real policy histories for a sealed pool

The scarce input is not cases but dated policy changes with real decisions under each version, because they supply unseen rules that people actually applied rather than rules written for a benchmark. Ask for the policy text by version, the effective dates, whether each version was ever published, and cases with outcomes on both sides of every change.

SourceX sources datasets to a buyer's specification rather than from stock: buyers describe the records they need, in what quantity, language and format, when, and under which license terms. A request that spells out the policy-level split, the first-public-date field and the acceptance test you will run lets a supplying company judge whether its records can support an unseen-policy evaluation before any agreement is drafted.

Frequently asked questions

Is an unseen policy test the same as a held-out test set?

No. A held-out test set keeps cases away from training, but those cases often run under rules the agent already practiced. An unseen policy test holds out the rules themselves, so the agent has to read and apply text it has not met before.

How many policies does an unseen policy evaluation need?

Enough that the policy-level result is not decided by one or two rules. There is no universal number: estimate how much per-policy scores vary on development policies and size the test pool from that spread. Always print the number of policies next to any average.

Can I use public benchmark policies as unseen policies?

Only with care. Text posted publicly can enter later training data; τ²-bench's policy files have been on GitHub since June 2025. Treat public policies as known, or edit them into counterfactual variants and label them as such.

What is a counterfactual policy in agent evaluation?

It is a rule written to contradict common practice while keeping the task's structure, so the correct action differs from what a model would assume. Research on counterfactual task variants found language model performance degraded substantially and consistently against the default versions of the same tasks.

Does fine-tuning on a benchmark's train split leak the test policy?

If both splits run under the same policy, yes: the agent has practiced the exact rules it is tested on. That is acceptable for measuring adherence to a known policy and invalid for claims about new ones. Split by policy whenever adaptation is the claim.

Sources

  • In τ²-bench, a domain environment reads the domain's policy file unless a different policy string is passed in. Source
  • τ²-bench's airline, retail and telecom folders each hold one policy file and task lists with train and test splits (50, 114 and 2,285 tasks at commit 267d161); the policy files were first committed on 2025-06-10. Source
  • τ²-bench version 0.2.1 (2025-11-07) added a Gymnasium-compatible interface for reinforcement learning training and train/test task splits for all domains. Source
  • SOP-Bench's benchmark data folders hold one procedure file each (one folder has two) with test files listing expected outputs: 15 procedure files and 2,154 rows at commit f6d4385. Source
  • SOP-Bench describes itself as built from more than 2,000 tasks across 12 industrial domains with human expert-authored procedures. Source
  • Counterfactual variants of 11 tasks showed nontrivial but substantially and consistently degraded language model performance compared with default conditions. Source
  • Llama Guard takes its safety taxonomy as input and can be adapted to other taxonomies through zero-shot or few-shot prompting. Source
  • OpenAI's gpt-oss-safeguard reasons over a developer-provided policy at inference time, so developers can revise policies without retraining. Source
  • Choosing analyses after seeing results on the same data voids the usual statistical assurances; the paper shows a holdout can be reused with protection. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies