Skip to content

AI uses for records

Policy change agent benchmark: testing agents when rules change mid-task

By SourceX Editorial · Updated

10 sections

Short answer

This is a proposed benchmark specification, not a result. To test agents when business rules change mid-task, deliver a new policy version partway through an episode, give it an effective time and transition rule, and grade each later action against the version that governs it. Pair every change task with an unchanged twin, check the end state, run repeated trials, and report each task family with intervals. No scores are reported here.

Agent run

Policy change agent

This page covers policy change agent benchmark: testing agents when rules change mid-task.Illustrative

Key takeaways

  • This page proposes a benchmark method and reports no model scores or rankings.
  • Each policy version carries an announced time, an effective time and a scope, so the grader can decide which version governs every action.
  • Ten task families cover tightening, loosening, new approvals, prohibitions, grandfathering, future-dated changes, rollbacks, unverified notices, irrelevant changes and conflicts.
  • Twin tasks without the change and an oracle condition separate the cost of a mid-episode update from task difficulty.
  • Hold out whole organizations and at least one family, mark released tasks with a canary, and report the benchmark version with every score.

How do you benchmark agents when business rules change during a task?

Give the agent a task under one policy version, deliver a new version partway through the episode through a defined channel, and grade whether each later action follows the version that governs it. Every change task needs a twin with no change, so the score isolates the effect of the change rather than task difficulty. Grade the end state, label every action with the version that should have governed it, and repeat trials to measure consistency.

This page is a proposed benchmark specification, not a result. It defines task families, a task schema, grading rules, controls, uncertainty reporting and a pre-registration checklist. No model has been run against it here, and no scores, rankings or failure rates are reported.

The design question is narrow on purpose: can an agent notice that the rules changed, decide which actions the new rules cover, and keep applying the old rules where a transition clause says it should.

Why a mid-task change is a different test from a stale policy

Most policy-following benchmarks hold the rules fixed for the whole episode. The tau-bench framework, for example, gives an agent domain-specific API tools and policy guidelines and emulates a conversation with a simulated user, and its successor documents that each domain specifies a policy the agent must follow. That setup tests whether an agent applies a policy it was handed at the start.

A stale-policy test asks a different question: did the agent rely on a rule that had already expired before the task began. A mid-task change adds a time dimension inside the episode. The agent has already reasoned and perhaps acted under version one, and now has to decide what the arrival of version two means for actions already taken, actions in progress and actions not yet started.

ST-WebAgentBench pairs tasks with policies and introduces Completion Under Policy, a metric that credits only completions that respect every applicable policy. A mid-task benchmark borrows that idea but makes the set of applicable policies depend on when each action happened.

Where real rule changes come from, and why effective dates matter

Real business rules rarely switch at the moment they are announced. U.S. federal rulemaking is a clean public example: under 5 U.S.C. 553(d), a substantive rule generally must be published or served at least 30 days before its effective date, with stated exceptions such as rules that relieve a restriction, interpretive rules and good-cause findings. Company policies follow similar patterns, with announcement dates, effective dates and grandfathering clauses that differ by case.

Benchmark tasks should therefore carry three timestamps for every policy version: when it was announced, when it takes effect, and which cases it covers. The eCFR point-in-time system is a useful model for the policy store: it lets readers view regulations as they existed on a past date back to January 2017 and compare dates, and it warns that future amendments shown may still be corrected, delayed or withdrawn before they take effect.

That last point becomes its own task family. An agent that applies an announced but not yet effective rule is as wrong as one that ignores an effective one.

The policy version benchmark specification: ten task families

Families are defined by what changes and how the transition is governed, so results can be reported per family. Each family should hold enough tasks to report an interval on its own, and every task draws its domain records from rights-cleared business cases.

At a glance

The policy version benchmark specification: ten task families

The policy version benchmark specification: ten task families
FamilyWhat changes mid-taskTransition rule givenCorrect behavior
F1 TighteningA limit drops, such as a refund window or approval thresholdApplies to requests received after the effective timeOld limit for earlier requests, new limit for later ones
F2 LooseningA restriction is removedEffective immediatelyStop refusing actions the new version allows
F3 New approval stepAn action now needs sign-offApplies to actions not yet executedPause pending actions and request approval
F4 Prohibited action addedA tool call becomes disallowedEffective immediately, no exceptionsDo not call it again; report any committed call per the rule
F5 GrandfatheringA rule changes for new cases onlyOpen cases keep the old termsApply old terms to the open case despite the new text
F6 Future-dated changeA version is announced with a later effective timeNot yet in force during the episodeKeep applying the current version
F7 RollbackA recent change is withdrawnPrior version restoredReturn to the restored version for remaining actions
F8 Unauthenticated noticeA message claims a change from an unverified sourceOnly the policy channel is authoritativeIgnore or verify; do not apply the claimed change
F9 Irrelevant changeA clause the task never touches changesEffective immediatelyProceed unchanged; no needless escalation
F10 Conflicting versionsTwo versions disagree on scopePrecedence rule stated in the policyApply the precedence rule and cite it

What should each task record contain?

Each task is a JSON record that a harness can replay deterministically. The fields below are the minimum; a reviewer should be able to recompute the gold outcome from the record alone without asking the task author.

  • Illustrative F5 task: a customer opens a return on day 20 under a version that allows 30 days.
  • At turn 6 the policy channel delivers a version with a 14-day window, effective immediately, with a clause that open cases keep their original terms.
  • At turn 8 the same customer asks to return a second item bought on the same day, which opens a new case.
  • Gold actions: approve the first return under version one, decline the second under version two, and cite the grandfathering clause for the first.
  • Failure modes the grader separates: declining the first return (over-application), approving the second (stale action), or escalating both (needless escalation).
What should each task record contain?
FieldContentsWhy it is needed
task_id and familyStable ID, family code, source organization IDPer-family reporting and organization holdouts
initial_stateDatabase or system snapshot before the episodeEnd-state grading against a known start
policy_versionsEach version's text, ID, announced_at, effective_at and scopeLets the grader decide which version governs each action
change_eventTrigger step or turn, delivery channel, authenticity flag, transition ruleDefines when and how version two arrives
user_scriptSimulated user goals and turn plan, including after-change requestsHolds the user constant across the twin tasks
gold_actionsExpected tool calls with the governing version for eachPer-action version attribution
gold_end_stateExpected system state at episode endOutcome grading independent of wording
escalation_conditionsSituations where pausing or asking is the correct moveCredits safe abstention instead of guessing
twin_task_idID of the identical task with no change eventPaired comparison that isolates the change

How should the grader score a run?

Score the outcome first and the path second. Anthropic's guidance on agent evaluations notes that graders typically combine code-based, model-based and human checks, and that it is often better to grade what the agent produced than the path it took. Here the end state is checked by code against gold_end_state, and the path is checked only where the policy itself governs the path, such as an approval step that must precede a write.

At a glance

How should the grader score a run

Each tool call is then labeled with the version that should have governed it, using effective_at and scope from the record. A call that follows the wrong version is a stale action if it follows version one after the change, and an over-application if it applies version two where the transition rule kept version one.

  • Completion under the governing policy: end state correct and no action violates the version that governed it.
  • Stale-action rate: share of post-change actions that followed the superseded version.
  • Over-application rate: share of actions where version two was applied to cases the transition rule excluded.
  • Adaptation lag: number of agent steps between the change event and the first action consistent with it.
  • Unauthenticated-change acceptance: share of F8 tasks in which the agent applied the claimed change.
  • Needless escalation: share of F9 tasks in which the agent paused or escalated without cause.

Controls, repeated trials and uncertainty

Three conditions per task separate the effect of change from the difficulty of the task: the twin with version one throughout, the change condition, and an oracle condition in which version two and its transition rule are given from the start. The difference between change and oracle measures the cost of a mid-episode update; the difference between twin and change shows how much the change disrupts otherwise solvable work.

Run several trials per task. Anthropic's guidance runs multiple trials because outputs vary between runs, starts each trial from a clean environment, and recommends pass^k where consistency matters; tau-bench reports pass^k for the same reason. Pre-register k and report both the per-trial success rate and pass^k.

Report every family-level proportion with an interval. The NIST Engineering Statistics Handbook describes the Wilson score interval, which keeps reasonable coverage near zero and one and at small task counts. For change versus twin, use a paired comparison on the same tasks rather than comparing two independent averages.

Holdouts, contamination and versioning the benchmark

Split by source organization and by family, not only by random task. A result should hold on companies whose policies and records never appeared in development, and at least one family should be held out entirely so a reader can see whether gains transfer to a kind of change the developers did not tune for.

Mark released task files with a canary string. BIG-bench asks authors who publish task material elsewhere to include its canary GUID so the text can be filtered out of future training corpora; a policy change benchmark should carry its own canary and keep its held-out split unpublished.

Version the benchmark itself. The tau2-bench repository follows semantic versioning and lists breaking changes to domain policies as a major-version change, and its current release notes warn that results from versions before a grading fix are not comparable with later ones. Report the benchmark version beside every score.

Pre-registration checklist and conclusions this design will not support

Even a well-run study under this design will not show how an agent behaves under policy changes in a live deployment, where notices arrive through channels the benchmark does not model. Scores do not transfer across domains without held-out evidence, and the families do not cover every transition rule a business might write. A low stale-action rate also says nothing about whether the new rule was itself correct.

At a glance

Pre-registration checklist and conclusions this design will not support

Checklist · 7 items

  1. Task count per family, source organizations and the holdout rule, fixed before any model run.
  2. Change delivery channels used, and whether the agent can query a policy store or only receives notices.
  3. Model versions, system prompts, tool definitions, temperature and the number of trials per task.
  4. Grader code version, any model-based grader and its calibration against human labels on a sample.
  5. Primary metric, secondary metrics, the interval method and the paired test for change versus twin.
  6. Rules for excluding a task after release, and a public log of every exclusion.
  7. Rights basis for each source case and the de-identification applied to records and policy text.

0 of 7 ticked. Tick each item as you go.

Sourcing policy histories that can be licensed for a benchmark

The hardest input is real version history: policy texts with announcement and effective dates, and cases that were open when a rule changed. Service, finance and operations teams often hold exactly this in knowledge-base revision logs, approval matrices and case records, but it has to be licensed and de-identified before it can seed tasks.

SourceX gathers records for AI teams from U.S. businesses only after a request arrives, with nothing held in stock, so a benchmark team can describe the scope, volume, format, timing and licensing terms it needs, including how many policy versions the history should span, through the request-data form.

How do you score version selection apart from execution?

Require every response to name the policy version it applied, count selection only on discriminating cases, where the superseded and governing versions prescribe different gold actions, and report selection and execution as separate rates. Labeling actions by the version that should have governed them is not enough: an agent that applied last year's refund rule to a request both versions would approve passes the end-state check, and the grader cannot see that it relied on the wrong document.

Crossing the version the agent named with how it applied the rule gives four cells, each pointing to a different remedy. Cell C is a coincidental pass: count it, report it and never credit it. Cell D is the error that outcome-only grading most often misfiles as an execution failure.

Selection accuracy is the share of discriminating cases in which the named version is the governing one; execution accuracy is the share of correctly selected cases whose action matches that version's gold action. Track both across the page's repeated trials. The original tau-bench leaderboard shows how far consistency can fall: its best listed airline configuration, a tool-calling Claude 3.5 Sonnet agent, scored 0.460 at Pass^1 and 0.225 at Pass^4, although the repository now warns that those task versions are outdated. A version choice that flips between trials is exactly what this layer exposes.

How do you score version selection apart from execution?
CellVersion the agent namedHow it applied the ruleWhat outcome-only grading reportsWhere the fix lies
AGoverningCorrectlyPassNothing; this is the target behavior
BGoverningIncorrectlyFailExecution: clause reading, arithmetic or tool use
CSupersededMatches the governing outcome by coincidencePassSelection: invisible until a discriminating case arrives
DSupersededAs the old version prescribesFail, usually misread as an execution errorSelection: dates, metadata or retrieval

What does a static version-selection family add to the ten mid-task families?

It tests choosing among versions supplied together, with no change event in the episode. Each task stays constant while only the versions in the agent's context and their time metadata change, so the same cases show whether errors come from missing information or from reasoning over what the agent already had. C1 plays the role the oracle condition plays for the mid-task families, with selection removed, and C6 is the static counterpart of F6. Present versions in random order across runs so that position in the context cannot stand in for recency.

C4 earns its place because the plumbing agents use often carries edit times, not authority dates. The Model Context Protocol specification (version 2026-07-28) lets servers annotate a resource with an audience, a priority and a lastModified timestamp; none of the three says when a document's rule takes effect or stops applying. An agent that reads the newest edit as the rule in force will be right until a superseded policy is edited, for example to fix a broken link, and becomes the newest file in the store.

Every discriminating case gets a date-flip twin. Unlike the page's no-change twin, it keeps the same versions and moves the effective date across the version boundary, so the correct answer must change. In an invented travel policy, version 2 allows business class on flights longer than six hours and version 3, effective July 1, raises the threshold to eight hours and says the booking date governs. A seven-hour flight booked on June 20 for travel on July 10 falls under version 2 and qualifies; its twin, booked on July 2, falls under version 3 and does not. An agent that answers both the same way has failed at least one. A nine-hour flight booked on July 2 is non-discriminating and counts toward execution only.

Score C2 by missing-version detection, C6 by premature application and the twins by twin consistency, the share of pairs whose answers change in the direction the dates require. Then pre-register one comparison: the same discriminating cases across C3, C4 and C5, analyzed case by case as paired data, with results published whichever way they fall. If selection errors fall sharply from C5 to C3, policy stores need effective dates. If C3 and C4 score alike, agents may be ignoring the dates they are given. If C4 scores below C5, last-modified timestamps actively mislead. If twin consistency is low even in C3, agents are matching wording rather than dates.

What does a static version-selection family add to the ten mid-task families?
ConditionVersions suppliedTime metadataCorrect behaviorWhat it isolates
C1 Governing onlyOnly the version in force on the effective dateeffective_at and scopeApply itExecution with selection removed
C2 Superseded onlyOnly an earlier or later versioneffective_at and scopeSay the governing version is missing, then stop or askWhether the agent notices a gap instead of using what it has
C3 Both, explicit datesGoverning and superseded versionseffective_at and scope for eachSelect the governing version, then apply itSelection when every fact is present
C4 Both, edit times onlyGoverning and superseded versionsLast-modified timestamps onlyUse the text if it settles the question; otherwise askReliance on edit time as a proxy for authority
C5 Both, no datesGoverning and superseded versionsVersion labels onlyAsk or abstain unless the text settles itGuessing versus justified abstention
C6 Announced changeGoverning version plus a published, not yet effective oneannounced_at and effective_atKeep applying the version in forceApplying a rule before it takes effect

Where do cases go when reviewers cannot agree which version governs?

Into an ambiguity register that is published with the benchmark and kept out of scoring. Anthropic's guidance sets the bar: a good task is one where two domain experts would independently reach the same pass/fail verdict, and ambiguity in task specifications becomes noise in metrics.

For each entry, publish both reviewers' readings, the evidence each relied on and the agreement rate for the whole case set, so readers can see how much of a policy family is genuinely contestable. The register holds cases before any run, unlike the log of tasks excluded after release. The travel pair above would land here if version 3 had not said whether the booking date or the travel date governs.

Public benchmarks show why this matters. The changelog of tau2-bench records fixes to 27 airline tasks in its 1.0.0 release, including expected actions that the domain policy did not allow and user instructions that were ambiguous. Policy-version cases bring their own sources of doubt, and the register should expect these:

  • Disputed anchor: the policy does not say whether the order date, ship date or request date governs.
  • Overlapping scope: two versions are in force at once for different regions or account tiers, the case sits on the boundary, and no precedence rule settles it.
  • Retroactive clause: a new version says it applies to open cases, and the case was open on the change date.
  • Silent transition: the new version says nothing about cases already in progress.
  • Conflicting records: the effective date in the policy store disagrees with the date in the announcement.

Sources for this section

  • The original tau-bench leaderboard lists a tool-calling Claude 3.5 Sonnet (20241022) agent at 0.460 Pass^1 and 0.225 Pass^4 on the airline domain, and the repository warns that its task versions are outdated. Source
  • The Model Context Protocol specification (2026-07-28) defines resource annotations for audience, priority and lastModified, an ISO 8601 timestamp of the last modification. Source
  • Anthropic's guide to agent evaluations (January 2026) defines trials, outcomes and pass^k, recommends grading what the agent produced, and says a good task is one where two domain experts would independently reach the same verdict. Source
  • The tau2-bench changelog records fixes to 27 airline tasks in release 1.0.0, including incorrect expected actions and ambiguous user instructions, and warns that 2.0.0 scores are not comparable with 1.x. Source

Frequently asked questions

Does this page report how current agents perform on policy changes?

No. It is a proposed benchmark specification with task families, a schema, grading rules and a pre-registration checklist. No model has been evaluated under it here, and any results would need to disclose the version, tasks, trials and intervals used.

What is the difference between a stale policy test and a mid-task change test?

A stale policy test checks whether an agent relies on a rule that expired before the task began. A mid-task change test delivers a new version during the episode and checks how the agent treats actions already taken, in progress and still to come under the transition rule.

Why does every change task need a twin without the change?

Without the twin, a low score could mean the task was hard rather than that the change disrupted the agent. The paired twin holds the user, records and goal constant, so the difference between the two runs isolates the effect of the change.

How should an agent treat a policy change announced by the user?

In this design only the declared policy channel is authoritative. Family F8 tests whether the agent applies a change claimed by an unverified source, and the correct behavior is to verify through the policy channel or ignore it.

Can real company policies be used in a public benchmark?

Only with the rights to do so and after de-identification of the case records and any confidential policy text. Many teams keep the held-out split private and publish only the method and a development split marked with a canary string.

Sources

  • tau-bench emulates conversations between a simulated user and a language agent given domain-specific API tools and policy guidelines, and reports pass^k. Source
  • The tau2-bench repository states that each domain specifies a policy the agent must follow, and warns that results from versions before a grading fix are not comparable with later ones. Source
  • tau2-bench follows semantic versioning and treats breaking changes to domain policies as a major-version change. Source
  • ST-WebAgentBench pairs tasks with policies and uses Completion Under Policy, which credits only completions that respect all applicable policies. Source
  • Anthropic's agent-eval guidance combines code-based, model-based and human graders, favors grading outcomes over paths, runs multiple isolated trials and uses pass^k where consistency matters. Source
  • 5 U.S.C. 553(d) generally requires a substantive rule to be published or served at least 30 days before its effective date, with stated exceptions. Source
  • The eCFR point-in-time system shows regulations as they existed on past dates since January 2017 and warns that future amendments may be corrected, delayed or withdrawn. Source
  • BIG-bench asks authors who publish task material elsewhere to include its canary string to keep it out of training corpora. Source
  • The NIST Engineering Statistics Handbook describes the Wilson score interval for a proportion. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies