Skip to content

AI uses for records

Agent evaluation for longitudinal drift: the same agent, three months later

By SourceX Editorial · Updated

10 sections

Short answer

To test the same agent after months of business drift, pin it (model snapshot, prompt, tools, index, settings), keep a frozen task set with a restorable environment, and run it beside a rolling set sampled from current work. Re-key part of the frozen set under today's rules. Comparing the arms separates agent change, workload-mix change and rule change, reported per task family with paired intervals.

Evaluation split

Agent evaluation

This page covers agent evaluation for longitudinal drift: the same agent, three months later.Illustrative

Key takeaways

  • An agent on a floating model name with edited prompts is not the same agent; record a fingerprint with every run.
  • Five drift types (agent, workload mix, ground truth, environment, grader) each need their own comparison.
  • Four arms (replay, re-keyed, rolling, production) turn one falling score into a decomposition you can act on.
  • Reweight rolling results to the baseline mix and report new kinds of work separately.
  • Use pass^k, paired differences and per-family thresholds set before the first rerun.

How do you test the same agent after several months of business drift?

Freeze a baseline and run it beside a rolling set. The baseline is a pinned agent (model snapshot, prompt, tools, retrieval index and settings), a versioned task set with answer keys, and an environment you can restore to the state the tasks were written against. The rolling set is a fresh, stratified sample of the work the business is doing now, keyed under the rules in force today.

Running the same pinned agent on both, and re-keying part of the frozen set under current rules, lets you say which of three things moved: the agent, the mix of work, or the business truth the answers depend on. A single score compared quarter to quarter cannot do that, because all three change at once and their effects can cancel out.

This matters because the word same is doing a lot of work. A study of the GPT-3.5 and GPT-4 API services found that the behavior of the same named service changed substantially between March and June 2023, which is why the authors called for continuous monitoring. An agent built on a floating model name, with prompts and tools edited by different teams, is rarely the same agent three months later unless someone pinned it.

Pin the agent before you measure it: the fingerprint record

Write down every component that can change the agent's behavior, and store it with each run. If any field differs between two runs, you are comparing two agents, and the report has to say so.

Model providers make this possible for the model itself. OpenAI's model pages, for example, describe dated snapshots as a way to lock in a specific model version so that performance and behavior stay consistent, while an alias can move to a newer snapshot. The rest of the fingerprint is your responsibility.

Pin the agent before you measure it: the fingerprint record
ComponentWhat to recordHow it changes unnoticedCheck before each run
ModelDated snapshot id, never an aliasAn alias is repointed to a newer snapshotSnapshot id in the run log matches the baseline
InstructionsHash of system prompt and few-shot examplesA team edits the prompt to fix one complaintHash equality
ToolsTool names, schemas and backend API versionsA field is renamed or a new required parameter appearsSchema diff against the stored copy
KnowledgeRetrieval index build id and document set versionNightly re-index adds new policy documentsIndex id equality, or a frozen index copy
MemoryMemory store snapshot or an empty-memory flagAccumulated memories steer later answersStart every trial from the stored state
SettingsTemperature, max steps, timeouts, retry policyA timeout is raised during an incident and never revertedConfig file hash
GraderGrader code version and any judge model snapshotThe LLM judge moves to a new modelJudge snapshot id and rubric hash

Which kind of drift are you looking at?

Separate five drift types before you design the runs, because each one needs a different comparison to show up. Mixing them is the most common reason a longitudinal report ends with a number nobody can act on.

Which kind of drift are you looking at?
Drift typeIllustrative business exampleComparison that isolates itWhat it hides if ignored
Agent driftA floating model name now points to a newer modelFingerprinted agent vs current agent on the same frozen tasksA provider update looks like a business change
Workload-mix driftWarranty claims rise from a small to a large share of ticketsRolling-set results reweighted to the baseline mixAverage score falls although no task family got worse
Ground-truth driftThe return window changes, so last quarter's correct answer is now wrongFrozen tasks scored under old keys and re-adjudicated new keysThe agent is blamed for following a retired rule, or praised for it
Environment driftA billing API adds a required field and old calls failFrozen tasks on the stored environment vs the live environmentIntegration breakage reported as reasoning failure
Grader driftThe judge model or rubric changes between runsRe-grade a stored transcript sample with both gradersScores move with no change in agent behavior

The frozen baseline and rolling evaluation protocol

The protocol runs four arms on a fixed cadence, usually monthly or after any known business change, and reports each against the baseline run made on the freeze date. Each arm answers one question; together they decompose the change.

Some environments cannot be restored: a third-party carrier API, a payment processor or a live inventory feed. In that case record the tool responses from the freeze-date run and replay them as fixtures for arm A, and label arm A as a mocked replay in the report. It still detects agent and grader drift, but not environment drift.

  • A minus its own freeze-date run: should be zero within noise. If not, stop and find the fingerprint field that moved before reading any other arm.
  • A minus B: the effect of rule changes on unchanged tasks, and the list of tasks whose correct answer flipped.
  • C reweighted to the baseline mix minus B: within-family change caused by new kinds of cases inside old families.
  • C unweighted minus C reweighted: the effect of the workload mix moving.
  • D minus C: the net value of every edit the team shipped since the freeze.
The frozen baseline and rolling evaluation protocol
ArmAgentTasks and environmentAnswer keyQuestion it answers
A. ReplayPinned fingerprintFrozen tasks on the restored environment snapshotOriginal keysIs the pinned agent still the agent we measured? Any change here is agent, grader or infrastructure drift
B. Re-keyedPinned fingerprintFrozen tasks on the restored snapshotKeys re-adjudicated under current rulesHow much does ground-truth drift alone cost the pinned agent?
C. RollingPinned fingerprintFresh stratified sample from the current window, live or freshly snapshotted environmentCurrent keysHow does the pinned agent do on today's work, by task family and after reweighting?
D. ProductionAgent as currently deployedSame rolling sample as arm CCurrent keysDid changes made since the freeze help or hurt on today's work?

What should the frozen baseline contain?

The frozen baseline should hold tasks that can be graded on the end state, an environment that can be put back exactly, and a coverage matrix that shows what is missing. The tau-bench design is a useful model: it scores an agent by comparing the database state at the end of a conversation with an annotated goal state, which makes a rerun months later gradable without a human reading every transcript.

At a glance

What should the frozen baseline contain

Restoring the environment is the part teams skip. Anthropic's guidance on agent evaluations says each trial should start from a clean environment and warns that leftover files, cached data and shared state cause correlated failures. For longitudinal work the clean environment must also be the old one: the account balances, open orders and policy documents as they stood on the freeze date.

  • Coverage matrix: task families as rows, severity tiers as columns, with a count in every cell and empty cells marked as known gaps.
  • Per task: the request, starting environment snapshot id, the governing rule and its version, the expected end state, required and forbidden actions, severity, and the date the key was adjudicated.
  • Severity rubric written by the business task owner, for example reversible inconvenience, customer-visible error, financial or contractual error, and safety or legal exposure.
  • Access controls: the frozen set lives where agent builders cannot tune against it; prompt fixes are tested on a separate development split.
  • A version manifest with the freeze date, item counts per cell and the fingerprint it was first run with.

Building the rolling set so it stays comparable

Sample the rolling set from a recent window of real work using the same task-family and severity strata as the baseline, then report it two ways: as observed and reweighted to the baseline's mix. The reweighting is ordinary direct standardization; it answers what today's score would be if the work looked like it did on the freeze date.

Work that fits no baseline family goes into a new-family stratum that is reported separately and never folded into the headline score. A new family is a finding in itself: it says the baseline no longer covers the business, and it is the main input to the next freeze.

Lag the window by the time it takes to verify an outcome. A refund task is only gradable once the payment has settled and any reversal has posted, and a dispatch task once the job is closed, so the rolling set should draw from work that finished, not work that started, in the window. Record the window bounds and the lag in the manifest so later cycles use the same rule.

Keep the rolling sample large enough per family to detect the change you care about. If a family has too few recent cases, pool two windows or report it as underpowered rather than printing a percentage built on a handful of tasks.

Is the decline real or noise?

Treat every arm as an experiment with uncertainty, compare paired results where tasks are shared, and set the alarm thresholds before the first rerun. Agents are stochastic, so a single trial per task overstates both improvements and declines.

The tau-bench authors introduced pass^k, the chance that all k independent trials of a task succeed, because consistency matters in customer-facing work; pass@k, by contrast, asks whether at least one of k attempts succeeds. For drift, pass^k is usually the more honest headline, since a task that now fails one run in three is a regression even if it still passes once.

Arms A and B share tasks with the freeze-date run, so analyze them as paired differences on the same items. A statistical treatment of language model evals recommends paired comparisons when models answer the same questions and clustered standard errors when questions fall into correlated groups; task families are exactly such groups. NIST's AI RMF playbook asks organizations to decide how much distributional shift or drift from baseline performance is acceptable, which is the threshold this protocol needs written down in advance.

Narrow the claim, or stop, when two conditions hold together: the environment cannot be restored or mocked, and nobody with authority can re-adjudicate the answer keys under current rules. Without those, the protocol can report a monitoring signal, not a drift diagnosis. Bring in an evaluation statistician when the families are small or the thresholds will drive a release decision, and the business task owner whenever a key flips.

  • Report per family and severity tier: verified completion, pass^k, count of severe failures, and an interval for each.
  • Report coverage: share of current work that falls in baseline families, and the size of the new-family stratum.
  • Set two thresholds per family: a watch level that triggers transcript review and an action level that triggers a fix or a re-freeze.
  • Read a sample of failing transcripts every cycle; a score change without a readable cause is not yet a finding.

Worked example (illustrative): a returns agent before and after a policy change

All figures in this example are illustrative, invented to show how the arms read; they are not measurements. A retailer froze a returns-and-refunds agent in July with 240 frozen tasks across six families. In September the business shortened the return window for one product line and added a new store-credit option.

Reading the arms in order: replay shows no change, so the pinned agent and grader are intact. Re-keying flips the correct answer on 18 frozen tasks, and the pinned agent gets most of those wrong because it still applies the old window. The rolling set shows a larger drop unweighted than reweighted, because exchange requests, a harder family, now make up more of the work. Production does better than the pinned agent on the rolling set because the team updated the policy documents in August.

Worked example (illustrative): a returns agent before and after a policy change
ArmIllustrative pass^3Change vs freezeReading
Freeze-date run82%baselineReference point for all comparisons
A. Replay81%within noiseNo agent, grader or infrastructure drift
B. Re-keyed75%lowerRule change alone explains most of the loss on old tasks
C. Rolling, reweighted74%lowerWithin-family change is close to the rule effect
C. Rolling, observed70%lower againMix shift toward exchanges adds a further drop
D. Production79%partly recoveredAugust document update fixed most rule errors

Which business task or failure type is missing from the drift test?

Check the coverage matrix against the work the business does now, and against the ways the agent can fail that an end-state check cannot see. The gaps below are the ones longitudinal evaluations most often leave out.

  • Tasks whose right answer is to escalate or refuse, which drift when policies add new exceptions.
  • Rare, high-severity cases, such as refunds above an approval limit or requests on accounts under dispute; sample them deliberately.
  • Multi-system tasks where one backend changed and another did not, which surface as environment drift.
  • Silent failures that leave the end state correct but tell the customer something false; add a communication check.
  • Work handled by a new channel or product line, which appears only in the new-family stratum.

Dated work records as the raw material for rolling sets

A rolling set is only as good as the stream of recent, dated work behind it, and an agent builder evaluating a business workflow often lacks that stream. SourceX does not hold such streams in inventory; it looks for U.S. businesses that keep them once a buyer describes, through the request form, the task families, windows, volume and format a rolling set needs, along with the licensing terms. Around those streams, SourceX takes care of rights review, contracting, preparation and delivery.

Frequently asked questions

How often should the frozen baseline be rerun?

Rerun it on a fixed cadence, monthly for fast-changing workflows such as support and quarterly for slow ones, and also immediately after any known change to the model, tools, policies or source systems. The fixed cadence gives a trend line; the event-driven runs tell you which change caused a step.

Can I compare this quarter's production metrics with last quarter's instead?

Production metrics are worth tracking, but they mix agent changes, workload mix and rule changes into one number. Use them as the trigger to run the protocol, not as the diagnosis.

What if the provider retires the model snapshot I pinned?

Run the new snapshot and the retiring one side by side on the frozen set while both are available, and record the bridge result. After retirement, treat the new snapshot as a new baseline fingerprint and keep the bridge in the version manifest.

Should failing rolling-set tasks be added to the frozen baseline?

Not mid-cycle, because it changes what the baseline measures. Collect them for the next freeze, where they can be adjudicated, assigned to a family and counted in the coverage matrix.

Is an LLM judge safe to use across several months?

Only if the judge is pinned to a snapshot and its rubric is versioned like any other fingerprint field. Re-grade a stored sample of transcripts whenever either changes, and report the difference as grader drift.

Sources

  • Anthropic's agent evaluation guide defines pass@k and pass^k, says each trial should start from a clean environment because shared state causes correlated failures, describes regression evals, and says production monitoring detects distribution drift after launch. Source
  • Chen, Zaharia and Zou found that the behavior of the same named GPT-3.5 and GPT-4 API services changed substantially between March and June 2023 and called for continuous monitoring. Source
  • The tau-bench paper grades agents by comparing the final database state with an annotated goal state and introduces pass^k, the chance that all k independent trials of a task succeed. Source
  • Adding Error Bars to Evals treats evals as experiments and recommends paired comparisons on shared questions and clustered standard errors when questions fall into correlated groups. Source
  • The NIST AI RMF playbook's Measure guidance covers monitoring AI system functionality and behavior in production and asks how much distributional shift or drift from baseline performance is acceptable. Source
  • OpenAI's model documentation says snapshots let you lock in a specific model version so that performance and behavior remain consistent, while aliases can point to newer snapshots. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies