Skip to content

AI uses for records

Grader policy versioning: revalidating a grader after a policy change

By SourceX Editorial · Updated

10 sections

Short answer

To revalidate an agent grader after a policy change, diff the policy clause by clause, map each changed clause to the grading criteria and tasks that encode it, regenerate those expected outcomes, then re-grade frozen past trajectories with old and new graders. Every verdict that flips must trace to the policy diff. Run shortcut probes before sign-off, and report scores from different policy versions as separate series.

Grader rubric

Grader policy versioning

This page covers grader policy versioning: revalidating a grader after a policy change.Illustrative

Key takeaways

  • In most agent benchmarks the policy lives in the expected outcomes, so a policy change can make a grader wrong without any grading code changing.
  • A policy to criterion version map links each clause version to the criteria, grader types and tasks that inherit it.
  • Re-grading frozen trajectories with old and new definitions isolates grader changes from agent changes; every flip needs an explanation.
  • The date rule that decides which policy version applies causes more errors than the new threshold itself.
  • Inaction, old-policy, phrase-only and over-grant probes catch graders that reward the wrong signal after a change.
  • Scores produced under different policy versions are different measurements and belong in separate series.

How do you revalidate an agent grader after a policy change?

Treat the policy change as a change to the answer key, not to the agent. Diff the old and new policy clause by clause, find every grading criterion and every task whose expected outcome depends on a changed clause, regenerate those expected outcomes under the new version, and then re-grade a frozen set of past trajectories with both the old and the new grader. Every pass or fail that flips must be explained by the policy diff; anything else is a grader bug.

Two outputs make the work auditable: a policy to criterion version map that says which clause version each criterion encodes, and a flip report that lists every verdict that changed and why. Scores produced under different policy versions are then labeled as different measurements and never averaged into one trend line.

This page covers the grader side of that problem for AI labs and evaluation teams building customer service and operations agents. It assumes you already have tasks, a policy document the agent must follow and some mix of state checks, string checks and model-based judges.

Why can a grader go stale when no grading code changed?

Because in most agent benchmarks the policy lives inside the expected answers, not inside the scoring function. In the open tau2-bench framework from Sierra, a task's final reward is the product of the components listed in its reward basis, which by default is a database check and a communication check. The expected database state is produced by replaying one reference trajectory on a fresh environment, and the communication check is a substring match on strings the agent must say. Neither component reads the policy at grading time. If the policy now allows a refund the old policy refused, the replayed reference still encodes a refusal, and a correct agent fails.

The airline policy that ships with the benchmark shows how fine the dependency is. It allows a cancellation only when the booking was made within the last 24 hours, the airline cancelled the flight, the ticket is in business class, or travel insurance covers the stated reason, and it warns that the cancellation API does not check those rules itself. Change any one of those conditions and some set of tasks now has a wrong target state.

This is not hypothetical. The project's changelog lists fixes to 27 airline tasks whose expected actions contradicted the written policy, for example compensation the policy did not allow and cancellations that did not meet the criteria. Its 1.0.1 release, dated July 15, 2026, corrected a gold refund value in a banking task and states that banking scores from before and after the release must not be compared. Model-based graders carry the same exposure: in the UK AI Security Institute's Inspect framework, the criterion a model grader sees is filled from each sample's target field, so a stale target produces a confidently stale verdict.

The policy to criterion version map

The map is a register with one row per pairing of a policy clause version and a grading criterion version. It answers three questions for any score: which policy text was in force, which criterion encoded it, and which tasks inherit that criterion. Keep it in the same repository as the tasks so a policy edit and a criterion edit land in one reviewable change.

The fields below are the minimum that let a reviewer trace a failing verdict back to a sentence of policy. Example values are illustrative.

The policy to criterion version map
FieldWhat to recordIllustrative value
Policy id and versionDocument name, semantic version and the date the version takes effectreturns-policy 3.0.0, effective 2026-09-01
Clause id and text hashStable clause identifier plus a hash of its normalized text, so silent edits are detectableRET-4.2, hash of the window sentence
Criterion id and versionThe grading rule that encodes the clause, versioned separately from the policyCRIT-RET-WINDOW v4
Grader typeState check, required-string check, action check or model judge with its rubric versionState check on order status plus judge rubric v2 for the explanation
Anchor date ruleWhich date decides the applicable version: order date, request date or decision dateOrder date
Tasks inheriting the criterionTask ids whose expected outcome depends on this clauseSeven return tasks (illustrative count)
Gold derivationHow the expected outcome is produced: replayed reference, hand-written assertion or adjudicated labelReplayed reference trajectory
Change classWording, threshold, new exception, removed permission, new disclosure or anchor-date changeThreshold
Revalidation statusNot started, gold regenerated, flips reviewed, signed offFlips reviewed
Adjudicator and dissentWho signed off and any recorded disagreement with the final readingPolicy owner signed; one evaluator dissent logged

Which kinds of policy change break which graders?

Classify the diff before touching any task, because the class decides how much re-grading is needed. A wording change that keeps meaning usually needs only a rubric review for model judges. A threshold change, such as a shorter window, invalidates every state target near the boundary. A new exception or a removed permission flips outcomes for whole task families and often needs new tasks that exercise the new branch.

Which kinds of policy change break which graders?
Change classState and action checksRequired-string checksModel judge rubrics
Wording only, same meaningUsually unaffectedAffected if the required phrase was quoted from the old textRe-read rubric; spot check verdicts
Threshold or limit changedTargets near the boundary are wrong until regeneratedAffected if amounts or durations are quotedRubric must cite the new value
New exception addedTasks that now qualify have wrong refusal targetsMay need a new required disclosureRubric needs the exception and a test case
Permission removedTasks that used it now expect a refusal or transferOld confirmation phrases may now be wrongJudge must penalize the removed action
New disclosure requiredUsually unaffectedAdd the disclosure; check substring matching is not trivially gamedRubric adds the disclosure as a criterion
Anchor date rule changedEvery task that spans the effective date is suspectAffected if dates are statedRubric must state which date governs

How do you re-grade old trajectories without re-running the agent?

Freeze a set of recorded trajectories from before the change and grade them twice, once with the old task definitions and once with the new ones. Both reference tools support this. Inspect's score command re-scores an existing log, optionally with a different scorer, and by default writes a new log file with a scored suffix rather than overwriting the original. tau2-bench added a fresh-tasks option in 1.0.1 that re-grades trajectories against the current task definitions instead of the copies embedded in each results file.

Sort the output into four cells: pass under both, fail under both, pass to fail and fail to pass. Only the two flip cells need human review, and each flip should be traceable to a row in the version map. A flip on a task that inherits no changed criterion is the most valuable finding in the exercise, because it means some other part of the grading changed too.

Keep the agent and its prompt constant while you do this. If the agent's system prompt also switched to the new policy text, the agent's behavior and the grader's targets moved together and you cannot tell which change produced a flip.

Which shortcut could raise the score without meeting the new requirement?

Look for any path where the grader rewards a visible signal instead of the business outcome. Google DeepMind describes specification gaming as behavior that exploits a loophole in the task specification at the expense of the intended outcome, and policy changes open fresh loopholes because the grader is usually updated in a hurry. Anthropic's guidance on agent evals makes the same point from the outcome side: an agent saying a flight is booked matters less than whether the reservation exists in the environment's database.

At a glance

Which shortcut could raise the score without meeting the new requirement

tau2-bench's own documentation gives a clean example of the risk. In one airline task the correct behavior is to refuse a cancellation; the reference actions are read-only, so the target database equals the initial one, and an agent that does nothing but politely refuse receives full reward. That is correct under that policy. If a later policy makes the same request eligible and the target is not regenerated, a do-nothing agent keeps its full score and a compliant agent is penalized.

Run these adversarial probes against every revalidated criterion before sign-off:

Checklist · 6 items

  1. Inaction probe: an agent that refuses everything. It should fail every task where the new policy grants the request.
  2. Old-policy probe: an agent that follows the previous version faithfully. It should fail exactly the tasks the version map lists as changed.
  3. Phrase-only probe: an agent that says each required string but takes no action. Substring checks alone should never pass it.
  4. Over-grant probe: an agent that approves every request. It should fail every task the new policy still refuses.
  5. Boundary probe: tasks placed one unit either side of a changed threshold, so an off-by-one in the target shows up.
  6. Judge probe: a long, confident explanation of the wrong decision, to test whether a model judge rewards fluency over the decision itself.

0 of 6 ticked. Tick each item as you go.

Worked example: a return window shortened from 30 to 14 days

Illustrative case. A retail support benchmark has 120 tasks. The returns policy moves from version 2.3 to 3.0, shortening the standard return window from 30 days to 14 days for orders placed on or after the effective date, while keeping 30 days for earlier orders. The version map shows that clause RET-4.2 feeds one state criterion and one judge rubric, inherited by 11 tasks.

The anchor date rule matters more than the threshold. Four of the 11 tasks involve orders placed before the effective date, so their targets do not change at all; a team that regenerated every return task against the new window would have introduced four new errors. Five tasks fall between day 15 and day 30 after a post-change order and flip from approve to refuse. Two tasks sit inside 14 days and keep their targets.

Re-grading 600 frozen trajectories (illustrative, five per task) produces 22 flips. Nineteen trace to the five changed tasks. Three come from an unrelated task whose required string quoted the old window in a confirmation message, a dependency nobody had recorded. That row is added to the map, the string is regenerated, and the old and new leaderboards are published as separate series rather than one trend.

How do you keep expert disagreement instead of averaging it away?

Record disagreement as data. New policy text is often ambiguous at exactly the cases that matter, and two qualified reviewers can read an exception differently. When that happens, the version map should hold both readings, the policy owner's ruling, and the tasks affected, and the ambiguous tasks should be either rewritten to remove the ambiguity or tagged so their scores can be reported separately.

Model judges need the same calibration. Zheng and colleagues found that strong LLM judges reached over 80% agreement with human preferences, about the same level humans reach with each other, which also means a judge will disagree with your adjudicators on a meaningful share of hard cases. Anthropic's guidance on agent evals describes teams moving from manual grading to LLM graders whose criteria are set by the product team and checked by periodic human calibration. After a policy change, run that calibration on the changed tasks first, with the adjudicated labels as the reference.

Do not resolve disagreement by majority vote among judges that were all prompted with the same stale rubric. Agreement among them measures consistency, not correctness.

When should scores stop being compared across versions?

Whenever the change alters what counts as a correct outcome. tau2-bench's versioning policy reserves a major version bump for breaking changes to domain policies, and its 1.0.1 notes explicitly forbid comparing affected scores across the release. Adopt the same rule internally: a policy change that flips any target starts a new score series, and charts show a break rather than a line.

Stop and escalate instead of patching when the version map cannot name a single clause behind a flip, when the policy owner and the domain adjudicator cannot agree on a reading, or when the changed clause depends on law or a contract the evaluation team cannot interpret. Narrow the benchmark by retiring the affected tasks until a ruling exists; a smaller trustworthy suite beats a larger one with known wrong targets.

Sourcing evaluation tasks that carry their policy history

Revalidation is far easier when tasks come from records that already show which rule applied: tickets and case notes that cite a policy version, decision logs with effective dates, and the corrections a team made when an agent or a person applied an old rule. Real operating histories contain exactly the boundary cases and exceptions that synthetic tasks tend to miss.

Evaluation teams can ask SourceX to look for such histories at U.S. businesses; nothing is sold from a catalog, and the request-data form is where a team sets out scope, volume, language, format, timing and licensing terms, including the need for dated policy and decision records. Whichever business holds the records signs off on the scope in its agreement before a single record is released.

Frequently asked questions

What is grader policy versioning?

Grader policy versioning means recording which version of a business policy each grading criterion encodes, so every score can be traced to the policy text in force. It pairs policy clause versions with criterion versions and the tasks that inherit them. Without it, a policy change silently turns correct agents into failing ones.

Do I have to re-run the agent after a policy change?

Not to revalidate the grader. Re-grade frozen trajectories from before the change with the old and new task definitions and review every verdict that flips. Re-running the agent is a separate step that measures how the agent itself handles the new policy.

Can an LLM judge just read the new policy and grade correctly?

Sometimes, but it needs checking. A judge reads whatever criterion and rubric you give it, and its agreement with human reviewers is high but not perfect. Calibrate it on the changed tasks against adjudicated labels before trusting its verdicts.

How many tasks does a policy change usually affect?

It depends on how many tasks inherit the changed clause and on the date rule that decides which version applies. The version map answers this for your suite directly. Tasks spanning the effective date need individual review because the obvious target is often wrong.

Should old and new benchmark scores be shown on one chart?

Only with a visible break. When a policy change alters what counts as correct, scores before and after measure different things. Publish them as separate series and note the policy versions behind each.

Sources

  • The tau2-bench evaluation guide explains that a task's reward is the product of its reward-basis components, that the target database state comes from replaying one reference trajectory, that communication checks are substring matches, and that an agent that only refuses can earn full reward on a task whose correct answer is refusal. Source
  • The tau2-bench changelog lists fixes to 27 airline tasks whose expected actions contradicted the policy, and its 1.0.1 release (July 15, 2026) corrected a gold value, added a fresh-tasks re-grading option and states that affected scores across the release are not comparable. Source
  • tau2-bench's versioning policy reserves major version bumps for breaking changes, including breaking changes to domain policies. Source
  • The tau2-bench airline policy lists the four conditions under which a flight may be cancelled and states that the cancellation API does not check them. Source
  • Inspect's model-graded scorer fills the criterion shown to the grader model from each sample's target field and lets teams customize templates, instructions and grader models. Source
  • Inspect's score command re-scores an existing evaluation log, optionally with a different scorer, and by default writes a new log file with a scored suffix. Source
  • Anthropic's guide to agent evals defines graders as logic that scores an aspect of performance, stresses checking the environment outcome rather than the agent's claims, and describes LLM graders checked by periodic human calibration. Source
  • Zheng et al. (2023) report that strong LLM judges such as GPT-4 reach over 80% agreement with human preferences, the same level of agreement found between humans. Source
  • Google DeepMind describes specification gaming as an agent exploiting a loophole in the task specification at the expense of the intended outcome, caused by misspecification of the task. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies