Skip to content

AI uses for records

Long horizon agent plan revision data: testing replanning on new evidence

By SourceX Editorial · Updated

9 sections

Short answer

Test plan revision by planting a known evidence change at a fixed step, annotating in advance which planned steps it invalidates and which commitments must survive, then scoring the agent on detection, revision scope, preservation, response type and cited evidence. Pair each task with a control twin where nothing relevant changes, and check the end state over repeated runs.

Agent run

Long horizon agent plan

This page covers long horizon agent plan revision data: testing replanning on new evidence.Illustrative

Key takeaways

  • Plant one known evidence change at a fixed step and annotate the correct revision before any agent runs.
  • Score detection, revision scope, preservation of commitments, response type and cited evidence separately.
  • Control twins with no relevant change catch agents that rewrite plans at every message.
  • End state and repeated-run reliability remain the anchor, but they cannot show why a run failed.
  • Handoffs right after a trigger are where revised constraints and authority are most often lost.
  • Some evidence should not change the plan, including instructions embedded in tool output.

How do you test whether an agent revises its plan when evidence changes?

Plant a controlled change in the evidence at a known step of a task whose correct plan depends on it, write down in advance which planned steps that change invalidates and which commitments must survive, and score the agent against that annotation. Run a twin of the same task with no change, or with an irrelevant one, so an agent that rewrites its plan at every new message fails as clearly as one that ignores the change.

This makes plan revision measurable as five separate behaviors: noticing the change, revising the right steps, leaving the rest alone, keeping constraints and authority intact through the revision and explaining the revision by pointing to the evidence. A final outcome check still matters, but it comes last and does not stand in for any of them.

The method below is an annotation guide for evaluation and training data. It describes how to build and label the tasks; it reports no results for any model.

Why a final success score cannot tell good replanning from luck

An agent can reach the right end state without revising anything, and a well-revised plan can still fail on an unrelated step. Outcome-only grading folds both cases into one number, which hides exactly the behavior this test is meant to isolate.

Outcome grading is still the right anchor. The tau-bench paper grades tool-using agents by comparing the final database state with the expected one, and introduces pass^k, the chance that all k independent runs of a task succeed, because a single success says little about reliability. TravelPlanner separates the rate at which plans satisfy hard constraints from the final pass rate for plans that meet every constraint. Both ideas carry over: compare end state, rerun several times and report constraint satisfaction separately. The tau-bench repository reports Pass^1 through Pass^4 for each agent strategy, and AgentBoard, built on multi-round, partially observable environments, reports fine-grained progress rates rather than success alone.

Process claims also need testing rather than trust. The ReAct paper argues that interleaving reasoning with actions helps a model induce, track and update action plans and handle exceptions. An annotated trigger set is how you check whether a given agent actually updates its plan when a tool result contradicts it, instead of continuing on the original script.

Longer tasks make this more pressing. METR's work on measuring the length of tasks AI agents can complete defines a time horizon in terms of how long the tasks take human professionals, and longer tasks give the world more time to change underneath a plan.

Evidence triggers worth planting, including ones that should change nothing

Choose triggers from the ways real workflows change, and include at least one that should leave the plan untouched. The examples are illustrative and drawn from common business systems, not from any company's records.

Evidence triggers worth planting, including ones that should change nothing
Trigger typeIllustrative exampleCorrect revisionOver-revision trap
Tool result contradicts an assumptionInventory lookup shows the planned item is backorderedRepair the sourcing steps onlyRe-confirming the customer's order details from scratch
New requirement from a personCustomer moves the delivery date earlier in a ticket replyRe-sequence dependent steps to meet the new dateDropping earlier requirements the reply did not mention
Authority changeAn approver revokes a spending approval mid-taskPause the affected step and escalateContinuing under the old approval because it was granted earlier
Shared state changed by another actorA colleague reassigns the ticket in the help deskStop acting on the ticket and hand off notesOverwriting the colleague's changes
Action failedCalendar API rejects a bookingRetry with a documented limit, then choose an alternativeTreating the failed booking as done
Two sources disagreeCRM and billing system show different contract end datesFlag the conflict and resolve or escalate before actingSilently picking the newer-looking value
Irrelevant new evidenceA routine status email unrelated to the task arrivesNo change; note it if usefulRebuilding the plan around the new message
Time passesA promised callback window expires during the taskUpdate commitments and notify as policy requiresKeeping the expired window in the plan

The plan revision and evidence trigger annotation guide

Annotate each task once before any agent runs, then label each agent run against it. The gold fields describe what should happen; the observed fields describe what the agent did. Keep step identifiers stable so the two can be compared mechanically.

At a glance

The plan revision and evidence trigger annotation guide

Where the trigger sits matters as much as what it is. Place triggers early, in the middle and late in the task, and vary the distance between the step where a commitment is made and the step where the evidence threatens it; a revision that works three steps after a constraint was stated can fail thirty steps later. Spread tasks across every trigger type in the taxonomy rather than concentrating on the easiest to script, and give each trigger task its own control twin built from the same starting state.

Version everything that can drift: the task text, the tool responses, the policy the agent was given and the annotation itself. When a gold label changes after review, keep the earlier label and the reason, because scores computed against the old label are no longer comparable with new ones.

The plan revision and evidence trigger annotation guide
FieldGold or observedDefinitionAnnotator rule
trigger_id and trigger_stepGoldWhich change is injected and at which stepOne trigger per task unless the task tests compound changes
evidence_pointerGoldThe tool result, message or record that carries the changePoint to the exact item; the agent must be able to observe it
plan_beforeGoldNumbered steps of the correct plan before the triggerSteps small enough that each has one outcome
invalidated_stepsGoldSteps the change makes wrong or unnecessaryList step numbers; agree between two annotators
preserved_commitmentsGoldPromises, constraints and approvals that must surviveInclude ones stated far earlier in the task
expected_responseGoldnone, local repair, reorder, replan, escalate or stopChoose the least disruptive response that is still correct
detection_stepObservedFirst step where the agent acts on or cites the changeBlank if never detected
revised_stepsObservedSteps the agent changed, added or droppedMap to plan_before step numbers where possible
cites_evidenceObservedWhether the agent's stated reason points to the evidenceYes only if the cited item is the planted one
violations_afterObservedPreserved commitments broken after the triggerCount each broken commitment once

Scoring detection, scope and restraint separately

Score each behavior on its own scale, then combine only if a single number is truly needed. A combined score hides whether an agent is failing by inertia or by thrashing, and those call for different fixes.

Keep within-episode revision apart from learning across attempts. Reflexion, for example, has agents reflect on task feedback and keep that reflective text in an episodic memory buffer to do better on later trials. That is a different capability, so count only revisions made inside the run where the evidence appeared, and run each trial with fresh memory unless cross-trial learning is the thing being tested.

  • Detection rate: share of trigger tasks where detection_step is filled.
  • Detection delay: steps between trigger_step and detection_step, reported as a distribution, not just a mean.
  • Revision precision: share of revised_steps that appear in invalidated_steps.
  • Revision recall: share of invalidated_steps that the agent revised.
  • Preservation: share of preserved_commitments not broken after the trigger.
  • Restraint: share of control twins where the agent made no substantive change.
  • Response match: whether the agent's response type equals expected_response, with escalate and stop scored strictly.
  • End state and reliability: final state checked against the expected state, with the full task rerun several times and pass^k reported.

Which requirement, permission or state conflict can be lost at the next handoff?

Anything held only in one agent's working context can be lost at a handoff, and the most fragile items are the ones the trigger just changed. A revised deadline, a revoked approval or a known conflict between two systems is new, so it has not yet been written into the shared record the next agent or person will read.

Annotate handoffs as their own checkpoints. Anthropic's guidance on building agents says agents should get ground truth from the environment at each step, can pause for human feedback at checkpoints or when blocked, and are commonly given stopping conditions such as a maximum number of iterations. In a plan revision test, the handoff note is where that ground truth either survives or disappears.

Checklist · 6 items

  1. Active constraints, with the step where each was introduced.
  2. Authority currently held, who granted it and whether the trigger changed it.
  3. Commitments made to people outside the system, such as a promised time window.
  4. Open conflicts between sources and which one the plan currently relies on.
  5. Steps already executed that cannot be undone.
  6. The evidence that caused the latest revision, by pointer, so the receiver can check it.

0 of 6 ticked. Tick each item as you go.

Illustrative: a dispatch agent learns a part was substituted

This case is invented to show the annotation; the counts are illustrative. A dispatch agent plans a 24-step job: confirm the customer's time window, reserve a part, assign a technician with the right certification, book the visit, send the confirmation. At step 9 the warehouse lookup returns a substituted part that requires a different certification.

The gold annotation marks 3 steps invalidated (technician assignment, booking and the confirmation draft) and 2 commitments preserved (the customer's time window and a note that the site requires an escort). Expected response: local repair. The control twin returns the original part at step 9.

One run detects the change at step 10, reassigns the technician and keeps the time window, but rewrites the confirmation without the escort note: recall 3 of 3, preservation 1 of 2. A second run never cites the substitution and books the original technician: detection blank, revision recall 0 of 3. A third run handles the trigger correctly but, in the control twin, rebooks the customer anyway: it fails restraint. Each failure points to a different fix, which a single outcome score would not show.

When revising the plan is the wrong move

Not every new piece of evidence deserves a new plan. Revising is wrong when the evidence comes from an untrusted channel, when it conflicts with a more authoritative source or when acting on it would exceed the agent's authority. In those cases the correct response is to verify, flag or escalate, and the annotation should say so.

Tool output and inbound messages can also carry instructions that only look like evidence. Include at least one trigger where text inside a document or email tells the agent to change its plan, and mark expected_response as none or escalate, so the test rewards an agent that treats content as data.

Stop and bring in a domain expert before finalizing gold labels when the correct response depends on policy the annotators do not own, such as refund rules, safety procedures or approval limits. Measure agreement between two annotators on invalidated_steps and expected_response; where they disagree often, the task is underspecified and should be rewritten rather than scored.

Finding natural plan changes in operational records

Synthetic triggers prove the scoring works; real operational history shows how plans actually change. Change orders in project tools, reassigned and reopened tickets in help desks, rescheduled jobs in field service systems and amended purchase orders in ERPs each record a plan, the evidence that changed it and what happened next. Records like these, with timestamps and authorship intact, make triggers and gold revisions far more realistic than invented ones.

Using them requires more than access. Holding system access, having privacy permission to process the records, owning the business records and having a contractual right to license them are separate questions.

For agent teams that want this kind of change history, SourceX works request by request: it searches among U.S. businesses for the records described, rather than offering a ready-made catalog, and the request captures scope, volume, language, format, timing and licensing needs. A dispatch or project history reaches a buyer only after its owner has approved the scope and terms and signed.

What should a dataset request for plan-revision records ask suppliers to deliver?

Ask for linked change histories that fill the annotation fields above from real work: the state before a change, the evidence that changed it, what was revised and what was kept, who had authority, the reason written at the time and the outcome. Finished plans alone show what a good plan looks like, not when or why one should change.

Write the request from a failure you can already reproduce with the annotation guide. If agents keep acting on revoked approvals, make authority changes the required trigger type and ask for the revocation, the response and what followed; if they rebuild plans around irrelevant messages, ask for events that left the plan untouched. Run the acceptance column on the supplier's sample before licensing the rest.

What should a dataset request for plan-revision records ask suppliers to deliver?
Annotation field it feedsWhat the records must containAcceptance test on the sampleWhat is lost without it
plan_beforeThe plan, schedule or ticket state before the change, with step or task identifiers and a timestampRebuild the state as of the trigger time for 20 drawn cases using only fields that existed thenNo baseline, so no revision to score
trigger_step and evidence_pointerThe changed field, message or record, with old value, new value, time and source systemEvery drawn case shows old and new values, not only the final stateTriggers have to be invented
invalidated_stepsThe tasks, dates or assignments changed after the triggerEach later change links back to the trigger by ticket, order or change identifierRevision scope cannot be measured
preserved_commitmentsPromises and constraints recorded before the trigger, such as due dates, customer windows, approvals and site rulesEach commitment carries a creation time earlier than the triggerPreservation cannot be scored
expected_responseWho requested and approved the change, and the role or limit that allowed itApprover roles are present and pseudonymized consistentlyEscalation and local repair look the same
Revision reasonThe reason, comment or approval note written when the change was madeReason timestamps fall at or before the revision; none were added during exportReasons reflect hindsight
End stateFinal status and dates after the revision, such as fulfilled, missed or reopenedMost drawn cases reach an outcome inside the export windowA revision cannot be tied to success
Control twinsChanges and messages that arrived but left the plan untouched, drawn by the same rulePresent at the share stated in the requestNo real-data test of restraint

Which systems record the change, the reason and who approved it?

Systems that log field-level changes with time, author and previous value can supply most of the brief, but only for fields that were being logged during the period you want. Confirm what was recorded before agreeing scope, because history that was never logged cannot be exported later.

Three documented examples show the range. Jira's REST API returns a paginated list of all changelogs for an issue, sorted by date from the oldest. Zendesk describes ticket audits as a read-only history of a ticket's updates, each able to hold several events, and its change events carry the previous value. Microsoft's Dataverse auditing records who changed a record, when, which fields and their earlier values, but only where an administrator turned auditing on, and it deletes logs older than the configured retention period unless retention is set to forever.

These logs record what changed far more reliably than why. A reason field that was optional, or a comment written days later, fails the reason row of the brief, so ask for the share of drawn changes that carry a reason timestamped at or before the revision before pricing the full set. The list maps the trigger taxonomy above to the records that tend to carry each trigger.

  • Tool result contradicts an assumption: availability or status fields that change on an order or work order, with the old value kept in the history.
  • New requirement from a person: due-date or priority edits linked to the customer comment that asked for them.
  • Authority change: approval status edits with the approver and time, including revocations.
  • Shared state changed by another actor: assignee or owner changes in an issue or ticket history.
  • Time passes: service-level or due-date breaches recorded as events when they happened, not inferred later.
  • Irrelevant new evidence: comments and notifications on the same record that were followed by no change to plan fields.

When will plan-revision records not fix the agent?

When the agent never saw the change, could not reach the system holding it, or works in a process that never records the reason, more records will not help. Run the four checks below on the failing runs before writing the request.

Only failures that survive all four belong in the request, each with its trigger type, the step where it appeared and the end state you expect, so the supplier's sample can be scored with the same fields. No validated plan-revision study accompanies this guide; the brief is a method, and the first sample is where it gets tested. When the request goes through SourceX, which sources records on request rather than from stock, the trigger types and acceptance tests belong in the request's scope.

  • Observation: replay the failing runs and confirm the change appeared in a tool result the agent actually received. A cached or truncated response is a tooling fix.
  • Reach: confirm the agent had a tool and permission to query the system where the change lived. A revocation stored where the agent cannot read is an access gap.
  • Process: confirm the business records the change and its reason at all. Plans revised in meetings and never logged leave no trigger for an agent or a dataset to learn from.
  • Grading: confirm the expected revision is unambiguous. Anthropic's engineering team warns that ambiguity in task specifications becomes noise in metrics, and says reading transcripts shows whether a failure was a genuine agent mistake.

Sources for this section

  • Jira Cloud's REST API returns a paginated list of all changelogs for an issue, sorted by date and starting from the oldest. Source
  • Zendesk describes ticket audits as a read-only history of a ticket's updates, where each audit can contain several events. Source
  • Zendesk's ticket audit events reference lists change events, which include the previous value of the changed field. Source
  • Microsoft's Dataverse auditing records who changed a record, when, which fields and their previous values, is turned on by an administrator, and deletes logs older than the retention period unless retention is set to forever. Source
  • Anthropic's engineering post (January 9, 2026) says ambiguity in task specifications becomes noise in metrics and that transcripts show whether a failed task was a genuine agent mistake. Source

Frequently asked questions

What is an evidence trigger in agent evaluation?

It is a planned change in what the agent can observe, such as a tool result, a message or a record update, placed at a known step. Because the step and the change are known in advance, annotators can define the correct revision before any agent runs.

Why include tasks where nothing should change?

Without them, an agent that rewrites its plan at every new message looks responsive. Control twins measure restraint, which is as important as detection in long workflows where many inputs are irrelevant.

Is final task success enough to evaluate replanning?

No. End state checks remain essential, but an agent can succeed without revising or fail for unrelated reasons. Label detection, revision scope, preservation and response type separately, then report end state and repeated-run reliability alongside them.

How many annotators does a plan revision task need?

At least two for the gold fields, especially invalidated_steps and expected_response. Disagreement is a signal that the task or its policy is unclear and should be fixed before scoring agents.

Can real business records be used to build these tests?

Yes, when rights are in place. Change orders, reassigned tickets and rescheduled jobs show natural triggers and revisions. Confirm ownership, privacy permission and a license to use them for AI before they enter a test set.

Sources

  • ReAct (Yao et al.) interleaves reasoning traces with actions and argues the reasoning helps the model induce, track and update action plans and handle exceptions. Source
  • tau-bench evaluates tool-using agents by comparing the final database state with the annotated goal state and introduces pass^k, the probability that all k trials of a task succeed. Source
  • The tau-bench repository reports Pass^1 through Pass^4 for each agent strategy and directs users to its successor benchmark for updated tasks. Source
  • TravelPlanner evaluates planning agents on hard and commonsense constraints and reports a hard constraint pass rate and a final pass rate. Source
  • METR's paper measures AI agent capability as a time horizon defined by how long tasks take human professionals. Source
  • Reflexion agents verbally reflect on task feedback and keep reflective text in an episodic memory buffer to improve later trials. Source
  • Anthropic's guide says agents need ground truth from the environment at each step, can pause for human feedback at checkpoints, and often run with stopping conditions such as a maximum number of iterations. Source
  • AgentBoard evaluates agents in multi-round, partially observable environments and reports fine-grained progress rates. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies