AI uses for records
Agent evaluation for longitudinal drift: the same agent, three months later
By SourceX Editorial · Updated
10 sections
Short answer
To test the same agent after months of business drift, pin it (model snapshot, prompt, tools, index, settings), keep a frozen task set with a restorable environment, and run it beside a rolling set sampled from current work. Re-key part of the frozen set under today's rules. Comparing the arms separates agent change, workload-mix change and rule change, reported per task family with paired intervals.
Evaluation split
Agent evaluation
Key takeaways
- An agent on a floating model name with edited prompts is not the same agent; record a fingerprint with every run.
- Five drift types (agent, workload mix, ground truth, environment, grader) each need their own comparison.
- Four arms (replay, re-keyed, rolling, production) turn one falling score into a decomposition you can act on.
- Reweight rolling results to the baseline mix and report new kinds of work separately.
- Use pass^k, paired differences and per-family thresholds set before the first rerun.
How do you test the same agent after several months of business drift?
Freeze a baseline and run it beside a rolling set. The baseline is a pinned agent (model snapshot, prompt, tools, retrieval index and settings), a versioned task set with answer keys, and an environment you can restore to the state the tasks were written against. The rolling set is a fresh, stratified sample of the work the business is doing now, keyed under the rules in force today.
Running the same pinned agent on both, and re-keying part of the frozen set under current rules, lets you say which of three things moved: the agent, the mix of work, or the business truth the answers depend on. A single score compared quarter to quarter cannot do that, because all three change at once and their effects can cancel out.
This matters because the word same is doing a lot of work. A study of the GPT-3.5 and GPT-4 API services found that the behavior of the same named service changed substantially between March and June 2023, which is why the authors called for continuous monitoring. An agent built on a floating model name, with prompts and tools edited by different teams, is rarely the same agent three months later unless someone pinned it.
Pin the agent before you measure it: the fingerprint record
Write down every component that can change the agent's behavior, and store it with each run. If any field differs between two runs, you are comparing two agents, and the report has to say so.
Model providers make this possible for the model itself. OpenAI's model pages, for example, describe dated snapshots as a way to lock in a specific model version so that performance and behavior stay consistent, while an alias can move to a newer snapshot. The rest of the fingerprint is your responsibility.
| Component | What to record | How it changes unnoticed | Check before each run |
|---|---|---|---|
| Model | Dated snapshot id, never an alias | An alias is repointed to a newer snapshot | Snapshot id in the run log matches the baseline |
| Instructions | Hash of system prompt and few-shot examples | A team edits the prompt to fix one complaint | Hash equality |
| Tools | Tool names, schemas and backend API versions | A field is renamed or a new required parameter appears | Schema diff against the stored copy |
| Knowledge | Retrieval index build id and document set version | Nightly re-index adds new policy documents | Index id equality, or a frozen index copy |
| Memory | Memory store snapshot or an empty-memory flag | Accumulated memories steer later answers | Start every trial from the stored state |
| Settings | Temperature, max steps, timeouts, retry policy | A timeout is raised during an incident and never reverted | Config file hash |
| Grader | Grader code version and any judge model snapshot | The LLM judge moves to a new model | Judge snapshot id and rubric hash |
Which kind of drift are you looking at?
Separate five drift types before you design the runs, because each one needs a different comparison to show up. Mixing them is the most common reason a longitudinal report ends with a number nobody can act on.
| Drift type | Illustrative business example | Comparison that isolates it | What it hides if ignored |
|---|---|---|---|
| Agent drift | A floating model name now points to a newer model | Fingerprinted agent vs current agent on the same frozen tasks | A provider update looks like a business change |
| Workload-mix drift | Warranty claims rise from a small to a large share of tickets | Rolling-set results reweighted to the baseline mix | Average score falls although no task family got worse |
| Ground-truth drift | The return window changes, so last quarter's correct answer is now wrong | Frozen tasks scored under old keys and re-adjudicated new keys | The agent is blamed for following a retired rule, or praised for it |
| Environment drift | A billing API adds a required field and old calls fail | Frozen tasks on the stored environment vs the live environment | Integration breakage reported as reasoning failure |
| Grader drift | The judge model or rubric changes between runs | Re-grade a stored transcript sample with both graders | Scores move with no change in agent behavior |
The frozen baseline and rolling evaluation protocol
The protocol runs four arms on a fixed cadence, usually monthly or after any known business change, and reports each against the baseline run made on the freeze date. Each arm answers one question; together they decompose the change.
Some environments cannot be restored: a third-party carrier API, a payment processor or a live inventory feed. In that case record the tool responses from the freeze-date run and replay them as fixtures for arm A, and label arm A as a mocked replay in the report. It still detects agent and grader drift, but not environment drift.
- A minus its own freeze-date run: should be zero within noise. If not, stop and find the fingerprint field that moved before reading any other arm.
- A minus B: the effect of rule changes on unchanged tasks, and the list of tasks whose correct answer flipped.
- C reweighted to the baseline mix minus B: within-family change caused by new kinds of cases inside old families.
- C unweighted minus C reweighted: the effect of the workload mix moving.
- D minus C: the net value of every edit the team shipped since the freeze.
| Arm | Agent | Tasks and environment | Answer key | Question it answers |
|---|---|---|---|---|
| A. Replay | Pinned fingerprint | Frozen tasks on the restored environment snapshot | Original keys | Is the pinned agent still the agent we measured? Any change here is agent, grader or infrastructure drift |
| B. Re-keyed | Pinned fingerprint | Frozen tasks on the restored snapshot | Keys re-adjudicated under current rules | How much does ground-truth drift alone cost the pinned agent? |
| C. Rolling | Pinned fingerprint | Fresh stratified sample from the current window, live or freshly snapshotted environment | Current keys | How does the pinned agent do on today's work, by task family and after reweighting? |
| D. Production | Agent as currently deployed | Same rolling sample as arm C | Current keys | Did changes made since the freeze help or hurt on today's work? |
What should the frozen baseline contain?
The frozen baseline should hold tasks that can be graded on the end state, an environment that can be put back exactly, and a coverage matrix that shows what is missing. The tau-bench design is a useful model: it scores an agent by comparing the database state at the end of a conversation with an annotated goal state, which makes a rerun months later gradable without a human reading every transcript.
At a glance
What should the frozen baseline contain
Restoring the environment is the part teams skip. Anthropic's guidance on agent evaluations says each trial should start from a clean environment and warns that leftover files, cached data and shared state cause correlated failures. For longitudinal work the clean environment must also be the old one: the account balances, open orders and policy documents as they stood on the freeze date.
- Coverage matrix: task families as rows, severity tiers as columns, with a count in every cell and empty cells marked as known gaps.
- Per task: the request, starting environment snapshot id, the governing rule and its version, the expected end state, required and forbidden actions, severity, and the date the key was adjudicated.
- Severity rubric written by the business task owner, for example reversible inconvenience, customer-visible error, financial or contractual error, and safety or legal exposure.
- Access controls: the frozen set lives where agent builders cannot tune against it; prompt fixes are tested on a separate development split.
- A version manifest with the freeze date, item counts per cell and the fingerprint it was first run with.
Building the rolling set so it stays comparable
Sample the rolling set from a recent window of real work using the same task-family and severity strata as the baseline, then report it two ways: as observed and reweighted to the baseline's mix. The reweighting is ordinary direct standardization; it answers what today's score would be if the work looked like it did on the freeze date.
Work that fits no baseline family goes into a new-family stratum that is reported separately and never folded into the headline score. A new family is a finding in itself: it says the baseline no longer covers the business, and it is the main input to the next freeze.
Lag the window by the time it takes to verify an outcome. A refund task is only gradable once the payment has settled and any reversal has posted, and a dispatch task once the job is closed, so the rolling set should draw from work that finished, not work that started, in the window. Record the window bounds and the lag in the manifest so later cycles use the same rule.
Keep the rolling sample large enough per family to detect the change you care about. If a family has too few recent cases, pool two windows or report it as underpowered rather than printing a percentage built on a handful of tasks.
Is the decline real or noise?
Treat every arm as an experiment with uncertainty, compare paired results where tasks are shared, and set the alarm thresholds before the first rerun. Agents are stochastic, so a single trial per task overstates both improvements and declines.
The tau-bench authors introduced pass^k, the chance that all k independent trials of a task succeed, because consistency matters in customer-facing work; pass@k, by contrast, asks whether at least one of k attempts succeeds. For drift, pass^k is usually the more honest headline, since a task that now fails one run in three is a regression even if it still passes once.
Arms A and B share tasks with the freeze-date run, so analyze them as paired differences on the same items. A statistical treatment of language model evals recommends paired comparisons when models answer the same questions and clustered standard errors when questions fall into correlated groups; task families are exactly such groups. NIST's AI RMF playbook asks organizations to decide how much distributional shift or drift from baseline performance is acceptable, which is the threshold this protocol needs written down in advance.
Narrow the claim, or stop, when two conditions hold together: the environment cannot be restored or mocked, and nobody with authority can re-adjudicate the answer keys under current rules. Without those, the protocol can report a monitoring signal, not a drift diagnosis. Bring in an evaluation statistician when the families are small or the thresholds will drive a release decision, and the business task owner whenever a key flips.
- Report per family and severity tier: verified completion, pass^k, count of severe failures, and an interval for each.
- Report coverage: share of current work that falls in baseline families, and the size of the new-family stratum.
- Set two thresholds per family: a watch level that triggers transcript review and an action level that triggers a fix or a re-freeze.
- Read a sample of failing transcripts every cycle; a score change without a readable cause is not yet a finding.
Worked example (illustrative): a returns agent before and after a policy change
All figures in this example are illustrative, invented to show how the arms read; they are not measurements. A retailer froze a returns-and-refunds agent in July with 240 frozen tasks across six families. In September the business shortened the return window for one product line and added a new store-credit option.
Reading the arms in order: replay shows no change, so the pinned agent and grader are intact. Re-keying flips the correct answer on 18 frozen tasks, and the pinned agent gets most of those wrong because it still applies the old window. The rolling set shows a larger drop unweighted than reweighted, because exchange requests, a harder family, now make up more of the work. Production does better than the pinned agent on the rolling set because the team updated the policy documents in August.
| Arm | Illustrative pass^3 | Change vs freeze | Reading |
|---|---|---|---|
| Freeze-date run | 82% | baseline | Reference point for all comparisons |
| A. Replay | 81% | within noise | No agent, grader or infrastructure drift |
| B. Re-keyed | 75% | lower | Rule change alone explains most of the loss on old tasks |
| C. Rolling, reweighted | 74% | lower | Within-family change is close to the rule effect |
| C. Rolling, observed | 70% | lower again | Mix shift toward exchanges adds a further drop |
| D. Production | 79% | partly recovered | August document update fixed most rule errors |
Which business task or failure type is missing from the drift test?
Check the coverage matrix against the work the business does now, and against the ways the agent can fail that an end-state check cannot see. The gaps below are the ones longitudinal evaluations most often leave out.
- Tasks whose right answer is to escalate or refuse, which drift when policies add new exceptions.
- Rare, high-severity cases, such as refunds above an approval limit or requests on accounts under dispute; sample them deliberately.
- Multi-system tasks where one backend changed and another did not, which surface as environment drift.
- Silent failures that leave the end state correct but tell the customer something false; add a communication check.
- Work handled by a new channel or product line, which appears only in the new-family stratum.
Dated work records as the raw material for rolling sets
A rolling set is only as good as the stream of recent, dated work behind it, and an agent builder evaluating a business workflow often lacks that stream. SourceX does not hold such streams in inventory; it looks for U.S. businesses that keep them once a buyer describes, through the request form, the task families, windows, volume and format a rolling set needs, along with the licensing terms. Around those streams, SourceX takes care of rights review, contracting, preparation and delivery.
Frequently asked questions
How often should the frozen baseline be rerun?
Rerun it on a fixed cadence, monthly for fast-changing workflows such as support and quarterly for slow ones, and also immediately after any known change to the model, tools, policies or source systems. The fixed cadence gives a trend line; the event-driven runs tell you which change caused a step.
Can I compare this quarter's production metrics with last quarter's instead?
Production metrics are worth tracking, but they mix agent changes, workload mix and rule changes into one number. Use them as the trigger to run the protocol, not as the diagnosis.
What if the provider retires the model snapshot I pinned?
Run the new snapshot and the retiring one side by side on the frozen set while both are available, and record the bridge result. After retirement, treat the new snapshot as a new baseline fingerprint and keep the bridge in the version manifest.
Should failing rolling-set tasks be added to the frozen baseline?
Not mid-cycle, because it changes what the baseline measures. Collect them for the next freeze, where they can be adjudicated, assigned to a family and counted in the coverage matrix.
Is an LLM judge safe to use across several months?
Only if the judge is pinned to a snapshot and its rubric is versioned like any other fingerprint field. Re-grade a stored sample of transcripts whenever either changes, and report the difference as grader drift.
Sources
- Anthropic's agent evaluation guide defines pass@k and pass^k, says each trial should start from a clean environment because shared state causes correlated failures, describes regression evals, and says production monitoring detects distribution drift after launch. Source
- Chen, Zaharia and Zou found that the behavior of the same named GPT-3.5 and GPT-4 API services changed substantially between March and June 2023 and called for continuous monitoring. Source
- The tau-bench paper grades agents by comparing the final database state with an annotated goal state and introduces pass^k, the chance that all k independent trials of a task succeed. Source
- Adding Error Bars to Evals treats evals as experiments and recommends paired comparisons on shared questions and clustered standard errors when questions fall into correlated groups. Source
- The NIST AI RMF playbook's Measure guidance covers monitoring AI system functionality and behavior in production and asks how much distributional shift or drift from baseline performance is acceptable. Source
- OpenAI's model documentation says snapshots let you lock in a specific model version so that performance and behavior remain consistent, while aliases can point to newer snapshots. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.