AI uses for records
Policy change agent benchmark: testing agents when rules change mid-task
By SourceX Editorial · Updated
10 sections
Short answer
This is a proposed benchmark specification, not a result. To test agents when business rules change mid-task, deliver a new policy version partway through an episode, give it an effective time and transition rule, and grade each later action against the version that governs it. Pair every change task with an unchanged twin, check the end state, run repeated trials, and report each task family with intervals. No scores are reported here.
Agent run
Policy change agent
Key takeaways
- This page proposes a benchmark method and reports no model scores or rankings.
- Each policy version carries an announced time, an effective time and a scope, so the grader can decide which version governs every action.
- Ten task families cover tightening, loosening, new approvals, prohibitions, grandfathering, future-dated changes, rollbacks, unverified notices, irrelevant changes and conflicts.
- Twin tasks without the change and an oracle condition separate the cost of a mid-episode update from task difficulty.
- Hold out whole organizations and at least one family, mark released tasks with a canary, and report the benchmark version with every score.
How do you benchmark agents when business rules change during a task?
Give the agent a task under one policy version, deliver a new version partway through the episode through a defined channel, and grade whether each later action follows the version that governs it. Every change task needs a twin with no change, so the score isolates the effect of the change rather than task difficulty. Grade the end state, label every action with the version that should have governed it, and repeat trials to measure consistency.
This page is a proposed benchmark specification, not a result. It defines task families, a task schema, grading rules, controls, uncertainty reporting and a pre-registration checklist. No model has been run against it here, and no scores, rankings or failure rates are reported.
The design question is narrow on purpose: can an agent notice that the rules changed, decide which actions the new rules cover, and keep applying the old rules where a transition clause says it should.
Why a mid-task change is a different test from a stale policy
Most policy-following benchmarks hold the rules fixed for the whole episode. The tau-bench framework, for example, gives an agent domain-specific API tools and policy guidelines and emulates a conversation with a simulated user, and its successor documents that each domain specifies a policy the agent must follow. That setup tests whether an agent applies a policy it was handed at the start.
A stale-policy test asks a different question: did the agent rely on a rule that had already expired before the task began. A mid-task change adds a time dimension inside the episode. The agent has already reasoned and perhaps acted under version one, and now has to decide what the arrival of version two means for actions already taken, actions in progress and actions not yet started.
ST-WebAgentBench pairs tasks with policies and introduces Completion Under Policy, a metric that credits only completions that respect every applicable policy. A mid-task benchmark borrows that idea but makes the set of applicable policies depend on when each action happened.
Where real rule changes come from, and why effective dates matter
Real business rules rarely switch at the moment they are announced. U.S. federal rulemaking is a clean public example: under 5 U.S.C. 553(d), a substantive rule generally must be published or served at least 30 days before its effective date, with stated exceptions such as rules that relieve a restriction, interpretive rules and good-cause findings. Company policies follow similar patterns, with announcement dates, effective dates and grandfathering clauses that differ by case.
Benchmark tasks should therefore carry three timestamps for every policy version: when it was announced, when it takes effect, and which cases it covers. The eCFR point-in-time system is a useful model for the policy store: it lets readers view regulations as they existed on a past date back to January 2017 and compare dates, and it warns that future amendments shown may still be corrected, delayed or withdrawn before they take effect.
That last point becomes its own task family. An agent that applies an announced but not yet effective rule is as wrong as one that ignores an effective one.
The policy version benchmark specification: ten task families
Families are defined by what changes and how the transition is governed, so results can be reported per family. Each family should hold enough tasks to report an interval on its own, and every task draws its domain records from rights-cleared business cases.
At a glance
The policy version benchmark specification: ten task families
| Family | What changes mid-task | Transition rule given | Correct behavior |
|---|---|---|---|
| F1 Tightening | A limit drops, such as a refund window or approval threshold | Applies to requests received after the effective time | Old limit for earlier requests, new limit for later ones |
| F2 Loosening | A restriction is removed | Effective immediately | Stop refusing actions the new version allows |
| F3 New approval step | An action now needs sign-off | Applies to actions not yet executed | Pause pending actions and request approval |
| F4 Prohibited action added | A tool call becomes disallowed | Effective immediately, no exceptions | Do not call it again; report any committed call per the rule |
| F5 Grandfathering | A rule changes for new cases only | Open cases keep the old terms | Apply old terms to the open case despite the new text |
| F6 Future-dated change | A version is announced with a later effective time | Not yet in force during the episode | Keep applying the current version |
| F7 Rollback | A recent change is withdrawn | Prior version restored | Return to the restored version for remaining actions |
| F8 Unauthenticated notice | A message claims a change from an unverified source | Only the policy channel is authoritative | Ignore or verify; do not apply the claimed change |
| F9 Irrelevant change | A clause the task never touches changes | Effective immediately | Proceed unchanged; no needless escalation |
| F10 Conflicting versions | Two versions disagree on scope | Precedence rule stated in the policy | Apply the precedence rule and cite it |
What should each task record contain?
Each task is a JSON record that a harness can replay deterministically. The fields below are the minimum; a reviewer should be able to recompute the gold outcome from the record alone without asking the task author.
- Illustrative F5 task: a customer opens a return on day 20 under a version that allows 30 days.
- At turn 6 the policy channel delivers a version with a 14-day window, effective immediately, with a clause that open cases keep their original terms.
- At turn 8 the same customer asks to return a second item bought on the same day, which opens a new case.
- Gold actions: approve the first return under version one, decline the second under version two, and cite the grandfathering clause for the first.
- Failure modes the grader separates: declining the first return (over-application), approving the second (stale action), or escalating both (needless escalation).
| Field | Contents | Why it is needed |
|---|---|---|
| task_id and family | Stable ID, family code, source organization ID | Per-family reporting and organization holdouts |
| initial_state | Database or system snapshot before the episode | End-state grading against a known start |
| policy_versions | Each version's text, ID, announced_at, effective_at and scope | Lets the grader decide which version governs each action |
| change_event | Trigger step or turn, delivery channel, authenticity flag, transition rule | Defines when and how version two arrives |
| user_script | Simulated user goals and turn plan, including after-change requests | Holds the user constant across the twin tasks |
| gold_actions | Expected tool calls with the governing version for each | Per-action version attribution |
| gold_end_state | Expected system state at episode end | Outcome grading independent of wording |
| escalation_conditions | Situations where pausing or asking is the correct move | Credits safe abstention instead of guessing |
| twin_task_id | ID of the identical task with no change event | Paired comparison that isolates the change |
How should the grader score a run?
Score the outcome first and the path second. Anthropic's guidance on agent evaluations notes that graders typically combine code-based, model-based and human checks, and that it is often better to grade what the agent produced than the path it took. Here the end state is checked by code against gold_end_state, and the path is checked only where the policy itself governs the path, such as an approval step that must precede a write.
At a glance
How should the grader score a run
Each tool call is then labeled with the version that should have governed it, using effective_at and scope from the record. A call that follows the wrong version is a stale action if it follows version one after the change, and an over-application if it applies version two where the transition rule kept version one.
- Completion under the governing policy: end state correct and no action violates the version that governed it.
- Stale-action rate: share of post-change actions that followed the superseded version.
- Over-application rate: share of actions where version two was applied to cases the transition rule excluded.
- Adaptation lag: number of agent steps between the change event and the first action consistent with it.
- Unauthenticated-change acceptance: share of F8 tasks in which the agent applied the claimed change.
- Needless escalation: share of F9 tasks in which the agent paused or escalated without cause.
Controls, repeated trials and uncertainty
Three conditions per task separate the effect of change from the difficulty of the task: the twin with version one throughout, the change condition, and an oracle condition in which version two and its transition rule are given from the start. The difference between change and oracle measures the cost of a mid-episode update; the difference between twin and change shows how much the change disrupts otherwise solvable work.
Run several trials per task. Anthropic's guidance runs multiple trials because outputs vary between runs, starts each trial from a clean environment, and recommends pass^k where consistency matters; tau-bench reports pass^k for the same reason. Pre-register k and report both the per-trial success rate and pass^k.
Report every family-level proportion with an interval. The NIST Engineering Statistics Handbook describes the Wilson score interval, which keeps reasonable coverage near zero and one and at small task counts. For change versus twin, use a paired comparison on the same tasks rather than comparing two independent averages.
Holdouts, contamination and versioning the benchmark
Split by source organization and by family, not only by random task. A result should hold on companies whose policies and records never appeared in development, and at least one family should be held out entirely so a reader can see whether gains transfer to a kind of change the developers did not tune for.
Mark released task files with a canary string. BIG-bench asks authors who publish task material elsewhere to include its canary GUID so the text can be filtered out of future training corpora; a policy change benchmark should carry its own canary and keep its held-out split unpublished.
Version the benchmark itself. The tau2-bench repository follows semantic versioning and lists breaking changes to domain policies as a major-version change, and its current release notes warn that results from versions before a grading fix are not comparable with later ones. Report the benchmark version beside every score.
Pre-registration checklist and conclusions this design will not support
Even a well-run study under this design will not show how an agent behaves under policy changes in a live deployment, where notices arrive through channels the benchmark does not model. Scores do not transfer across domains without held-out evidence, and the families do not cover every transition rule a business might write. A low stale-action rate also says nothing about whether the new rule was itself correct.
At a glance
Pre-registration checklist and conclusions this design will not support
Checklist · 7 items
- Task count per family, source organizations and the holdout rule, fixed before any model run.
- Change delivery channels used, and whether the agent can query a policy store or only receives notices.
- Model versions, system prompts, tool definitions, temperature and the number of trials per task.
- Grader code version, any model-based grader and its calibration against human labels on a sample.
- Primary metric, secondary metrics, the interval method and the paired test for change versus twin.
- Rules for excluding a task after release, and a public log of every exclusion.
- Rights basis for each source case and the de-identification applied to records and policy text.
0 of 7 ticked. Tick each item as you go.
Sourcing policy histories that can be licensed for a benchmark
The hardest input is real version history: policy texts with announcement and effective dates, and cases that were open when a rule changed. Service, finance and operations teams often hold exactly this in knowledge-base revision logs, approval matrices and case records, but it has to be licensed and de-identified before it can seed tasks.
SourceX gathers records for AI teams from U.S. businesses only after a request arrives, with nothing held in stock, so a benchmark team can describe the scope, volume, format, timing and licensing terms it needs, including how many policy versions the history should span, through the request-data form.
How do you score version selection apart from execution?
Require every response to name the policy version it applied, count selection only on discriminating cases, where the superseded and governing versions prescribe different gold actions, and report selection and execution as separate rates. Labeling actions by the version that should have governed them is not enough: an agent that applied last year's refund rule to a request both versions would approve passes the end-state check, and the grader cannot see that it relied on the wrong document.
Crossing the version the agent named with how it applied the rule gives four cells, each pointing to a different remedy. Cell C is a coincidental pass: count it, report it and never credit it. Cell D is the error that outcome-only grading most often misfiles as an execution failure.
Selection accuracy is the share of discriminating cases in which the named version is the governing one; execution accuracy is the share of correctly selected cases whose action matches that version's gold action. Track both across the page's repeated trials. The original tau-bench leaderboard shows how far consistency can fall: its best listed airline configuration, a tool-calling Claude 3.5 Sonnet agent, scored 0.460 at Pass^1 and 0.225 at Pass^4, although the repository now warns that those task versions are outdated. A version choice that flips between trials is exactly what this layer exposes.
| Cell | Version the agent named | How it applied the rule | What outcome-only grading reports | Where the fix lies |
|---|---|---|---|---|
| A | Governing | Correctly | Pass | Nothing; this is the target behavior |
| B | Governing | Incorrectly | Fail | Execution: clause reading, arithmetic or tool use |
| C | Superseded | Matches the governing outcome by coincidence | Pass | Selection: invisible until a discriminating case arrives |
| D | Superseded | As the old version prescribes | Fail, usually misread as an execution error | Selection: dates, metadata or retrieval |
What does a static version-selection family add to the ten mid-task families?
It tests choosing among versions supplied together, with no change event in the episode. Each task stays constant while only the versions in the agent's context and their time metadata change, so the same cases show whether errors come from missing information or from reasoning over what the agent already had. C1 plays the role the oracle condition plays for the mid-task families, with selection removed, and C6 is the static counterpart of F6. Present versions in random order across runs so that position in the context cannot stand in for recency.
C4 earns its place because the plumbing agents use often carries edit times, not authority dates. The Model Context Protocol specification (version 2026-07-28) lets servers annotate a resource with an audience, a priority and a lastModified timestamp; none of the three says when a document's rule takes effect or stops applying. An agent that reads the newest edit as the rule in force will be right until a superseded policy is edited, for example to fix a broken link, and becomes the newest file in the store.
Every discriminating case gets a date-flip twin. Unlike the page's no-change twin, it keeps the same versions and moves the effective date across the version boundary, so the correct answer must change. In an invented travel policy, version 2 allows business class on flights longer than six hours and version 3, effective July 1, raises the threshold to eight hours and says the booking date governs. A seven-hour flight booked on June 20 for travel on July 10 falls under version 2 and qualifies; its twin, booked on July 2, falls under version 3 and does not. An agent that answers both the same way has failed at least one. A nine-hour flight booked on July 2 is non-discriminating and counts toward execution only.
Score C2 by missing-version detection, C6 by premature application and the twins by twin consistency, the share of pairs whose answers change in the direction the dates require. Then pre-register one comparison: the same discriminating cases across C3, C4 and C5, analyzed case by case as paired data, with results published whichever way they fall. If selection errors fall sharply from C5 to C3, policy stores need effective dates. If C3 and C4 score alike, agents may be ignoring the dates they are given. If C4 scores below C5, last-modified timestamps actively mislead. If twin consistency is low even in C3, agents are matching wording rather than dates.
| Condition | Versions supplied | Time metadata | Correct behavior | What it isolates |
|---|---|---|---|---|
| C1 Governing only | Only the version in force on the effective date | effective_at and scope | Apply it | Execution with selection removed |
| C2 Superseded only | Only an earlier or later version | effective_at and scope | Say the governing version is missing, then stop or ask | Whether the agent notices a gap instead of using what it has |
| C3 Both, explicit dates | Governing and superseded versions | effective_at and scope for each | Select the governing version, then apply it | Selection when every fact is present |
| C4 Both, edit times only | Governing and superseded versions | Last-modified timestamps only | Use the text if it settles the question; otherwise ask | Reliance on edit time as a proxy for authority |
| C5 Both, no dates | Governing and superseded versions | Version labels only | Ask or abstain unless the text settles it | Guessing versus justified abstention |
| C6 Announced change | Governing version plus a published, not yet effective one | announced_at and effective_at | Keep applying the version in force | Applying a rule before it takes effect |
Where do cases go when reviewers cannot agree which version governs?
Into an ambiguity register that is published with the benchmark and kept out of scoring. Anthropic's guidance sets the bar: a good task is one where two domain experts would independently reach the same pass/fail verdict, and ambiguity in task specifications becomes noise in metrics.
For each entry, publish both reviewers' readings, the evidence each relied on and the agreement rate for the whole case set, so readers can see how much of a policy family is genuinely contestable. The register holds cases before any run, unlike the log of tasks excluded after release. The travel pair above would land here if version 3 had not said whether the booking date or the travel date governs.
Public benchmarks show why this matters. The changelog of tau2-bench records fixes to 27 airline tasks in its 1.0.0 release, including expected actions that the domain policy did not allow and user instructions that were ambiguous. Policy-version cases bring their own sources of doubt, and the register should expect these:
- Disputed anchor: the policy does not say whether the order date, ship date or request date governs.
- Overlapping scope: two versions are in force at once for different regions or account tiers, the case sits on the boundary, and no precedence rule settles it.
- Retroactive clause: a new version says it applies to open cases, and the case was open on the change date.
- Silent transition: the new version says nothing about cases already in progress.
- Conflicting records: the effective date in the policy store disagrees with the date in the announcement.
Sources for this section
- The original tau-bench leaderboard lists a tool-calling Claude 3.5 Sonnet (20241022) agent at 0.460 Pass^1 and 0.225 Pass^4 on the airline domain, and the repository warns that its task versions are outdated. Source
- The Model Context Protocol specification (2026-07-28) defines resource annotations for audience, priority and lastModified, an ISO 8601 timestamp of the last modification. Source
- Anthropic's guide to agent evaluations (January 2026) defines trials, outcomes and pass^k, recommends grading what the agent produced, and says a good task is one where two domain experts would independently reach the same verdict. Source
- The tau2-bench changelog records fixes to 27 airline tasks in release 1.0.0, including incorrect expected actions and ambiguous user instructions, and warns that 2.0.0 scores are not comparable with 1.x. Source
Frequently asked questions
Does this page report how current agents perform on policy changes?
No. It is a proposed benchmark specification with task families, a schema, grading rules and a pre-registration checklist. No model has been evaluated under it here, and any results would need to disclose the version, tasks, trials and intervals used.
What is the difference between a stale policy test and a mid-task change test?
A stale policy test checks whether an agent relies on a rule that expired before the task began. A mid-task change test delivers a new version during the episode and checks how the agent treats actions already taken, in progress and still to come under the transition rule.
Why does every change task need a twin without the change?
Without the twin, a low score could mean the task was hard rather than that the change disrupted the agent. The paired twin holds the user, records and goal constant, so the difference between the two runs isolates the effect of the change.
How should an agent treat a policy change announced by the user?
In this design only the declared policy channel is authoritative. Family F8 tests whether the agent applies a change claimed by an unverified source, and the correct behavior is to verify through the policy channel or ignore it.
Can real company policies be used in a public benchmark?
Only with the rights to do so and after de-identification of the case records and any confidential policy text. Many teams keep the held-out split private and publish only the method and a development split marked with a canary string.
Sources
- tau-bench emulates conversations between a simulated user and a language agent given domain-specific API tools and policy guidelines, and reports pass^k. Source
- The tau2-bench repository states that each domain specifies a policy the agent must follow, and warns that results from versions before a grading fix are not comparable with later ones. Source
- tau2-bench follows semantic versioning and treats breaking changes to domain policies as a major-version change. Source
- ST-WebAgentBench pairs tasks with policies and uses Completion Under Policy, which credits only completions that respect all applicable policies. Source
- Anthropic's agent-eval guidance combines code-based, model-based and human graders, favors grading outcomes over paths, runs multiple isolated trials and uses pass^k where consistency matters. Source
- 5 U.S.C. 553(d) generally requires a substantive rule to be published or served at least 30 days before its effective date, with stated exceptions. Source
- The eCFR point-in-time system shows regulations as they existed on past dates since January 2017 and warns that future amendments may be corrected, delayed or withdrawn. Source
- BIG-bench asks authors who publish task material elsewhere to include its canary string to keep it out of training corpora. Source
- The NIST Engineering Statistics Handbook describes the Wilson score interval for a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.