AI uses for records
Long horizon agent plan revision data: testing replanning on new evidence
By SourceX Editorial · Updated
9 sections
Short answer
Test plan revision by planting a known evidence change at a fixed step, annotating in advance which planned steps it invalidates and which commitments must survive, then scoring the agent on detection, revision scope, preservation, response type and cited evidence. Pair each task with a control twin where nothing relevant changes, and check the end state over repeated runs.
Agent run
Long horizon agent plan
Key takeaways
- Plant one known evidence change at a fixed step and annotate the correct revision before any agent runs.
- Score detection, revision scope, preservation of commitments, response type and cited evidence separately.
- Control twins with no relevant change catch agents that rewrite plans at every message.
- End state and repeated-run reliability remain the anchor, but they cannot show why a run failed.
- Handoffs right after a trigger are where revised constraints and authority are most often lost.
- Some evidence should not change the plan, including instructions embedded in tool output.
How do you test whether an agent revises its plan when evidence changes?
Plant a controlled change in the evidence at a known step of a task whose correct plan depends on it, write down in advance which planned steps that change invalidates and which commitments must survive, and score the agent against that annotation. Run a twin of the same task with no change, or with an irrelevant one, so an agent that rewrites its plan at every new message fails as clearly as one that ignores the change.
This makes plan revision measurable as five separate behaviors: noticing the change, revising the right steps, leaving the rest alone, keeping constraints and authority intact through the revision and explaining the revision by pointing to the evidence. A final outcome check still matters, but it comes last and does not stand in for any of them.
The method below is an annotation guide for evaluation and training data. It describes how to build and label the tasks; it reports no results for any model.
Why a final success score cannot tell good replanning from luck
An agent can reach the right end state without revising anything, and a well-revised plan can still fail on an unrelated step. Outcome-only grading folds both cases into one number, which hides exactly the behavior this test is meant to isolate.
Outcome grading is still the right anchor. The tau-bench paper grades tool-using agents by comparing the final database state with the expected one, and introduces pass^k, the chance that all k independent runs of a task succeed, because a single success says little about reliability. TravelPlanner separates the rate at which plans satisfy hard constraints from the final pass rate for plans that meet every constraint. Both ideas carry over: compare end state, rerun several times and report constraint satisfaction separately. The tau-bench repository reports Pass^1 through Pass^4 for each agent strategy, and AgentBoard, built on multi-round, partially observable environments, reports fine-grained progress rates rather than success alone.
Process claims also need testing rather than trust. The ReAct paper argues that interleaving reasoning with actions helps a model induce, track and update action plans and handle exceptions. An annotated trigger set is how you check whether a given agent actually updates its plan when a tool result contradicts it, instead of continuing on the original script.
Longer tasks make this more pressing. METR's work on measuring the length of tasks AI agents can complete defines a time horizon in terms of how long the tasks take human professionals, and longer tasks give the world more time to change underneath a plan.
Evidence triggers worth planting, including ones that should change nothing
Choose triggers from the ways real workflows change, and include at least one that should leave the plan untouched. The examples are illustrative and drawn from common business systems, not from any company's records.
| Trigger type | Illustrative example | Correct revision | Over-revision trap |
|---|---|---|---|
| Tool result contradicts an assumption | Inventory lookup shows the planned item is backordered | Repair the sourcing steps only | Re-confirming the customer's order details from scratch |
| New requirement from a person | Customer moves the delivery date earlier in a ticket reply | Re-sequence dependent steps to meet the new date | Dropping earlier requirements the reply did not mention |
| Authority change | An approver revokes a spending approval mid-task | Pause the affected step and escalate | Continuing under the old approval because it was granted earlier |
| Shared state changed by another actor | A colleague reassigns the ticket in the help desk | Stop acting on the ticket and hand off notes | Overwriting the colleague's changes |
| Action failed | Calendar API rejects a booking | Retry with a documented limit, then choose an alternative | Treating the failed booking as done |
| Two sources disagree | CRM and billing system show different contract end dates | Flag the conflict and resolve or escalate before acting | Silently picking the newer-looking value |
| Irrelevant new evidence | A routine status email unrelated to the task arrives | No change; note it if useful | Rebuilding the plan around the new message |
| Time passes | A promised callback window expires during the task | Update commitments and notify as policy requires | Keeping the expired window in the plan |
The plan revision and evidence trigger annotation guide
Annotate each task once before any agent runs, then label each agent run against it. The gold fields describe what should happen; the observed fields describe what the agent did. Keep step identifiers stable so the two can be compared mechanically.
At a glance
The plan revision and evidence trigger annotation guide
Where the trigger sits matters as much as what it is. Place triggers early, in the middle and late in the task, and vary the distance between the step where a commitment is made and the step where the evidence threatens it; a revision that works three steps after a constraint was stated can fail thirty steps later. Spread tasks across every trigger type in the taxonomy rather than concentrating on the easiest to script, and give each trigger task its own control twin built from the same starting state.
Version everything that can drift: the task text, the tool responses, the policy the agent was given and the annotation itself. When a gold label changes after review, keep the earlier label and the reason, because scores computed against the old label are no longer comparable with new ones.
| Field | Gold or observed | Definition | Annotator rule |
|---|---|---|---|
| trigger_id and trigger_step | Gold | Which change is injected and at which step | One trigger per task unless the task tests compound changes |
| evidence_pointer | Gold | The tool result, message or record that carries the change | Point to the exact item; the agent must be able to observe it |
| plan_before | Gold | Numbered steps of the correct plan before the trigger | Steps small enough that each has one outcome |
| invalidated_steps | Gold | Steps the change makes wrong or unnecessary | List step numbers; agree between two annotators |
| preserved_commitments | Gold | Promises, constraints and approvals that must survive | Include ones stated far earlier in the task |
| expected_response | Gold | none, local repair, reorder, replan, escalate or stop | Choose the least disruptive response that is still correct |
| detection_step | Observed | First step where the agent acts on or cites the change | Blank if never detected |
| revised_steps | Observed | Steps the agent changed, added or dropped | Map to plan_before step numbers where possible |
| cites_evidence | Observed | Whether the agent's stated reason points to the evidence | Yes only if the cited item is the planted one |
| violations_after | Observed | Preserved commitments broken after the trigger | Count each broken commitment once |
Scoring detection, scope and restraint separately
Score each behavior on its own scale, then combine only if a single number is truly needed. A combined score hides whether an agent is failing by inertia or by thrashing, and those call for different fixes.
Keep within-episode revision apart from learning across attempts. Reflexion, for example, has agents reflect on task feedback and keep that reflective text in an episodic memory buffer to do better on later trials. That is a different capability, so count only revisions made inside the run where the evidence appeared, and run each trial with fresh memory unless cross-trial learning is the thing being tested.
- Detection rate: share of trigger tasks where detection_step is filled.
- Detection delay: steps between trigger_step and detection_step, reported as a distribution, not just a mean.
- Revision precision: share of revised_steps that appear in invalidated_steps.
- Revision recall: share of invalidated_steps that the agent revised.
- Preservation: share of preserved_commitments not broken after the trigger.
- Restraint: share of control twins where the agent made no substantive change.
- Response match: whether the agent's response type equals expected_response, with escalate and stop scored strictly.
- End state and reliability: final state checked against the expected state, with the full task rerun several times and pass^k reported.
Which requirement, permission or state conflict can be lost at the next handoff?
Anything held only in one agent's working context can be lost at a handoff, and the most fragile items are the ones the trigger just changed. A revised deadline, a revoked approval or a known conflict between two systems is new, so it has not yet been written into the shared record the next agent or person will read.
Annotate handoffs as their own checkpoints. Anthropic's guidance on building agents says agents should get ground truth from the environment at each step, can pause for human feedback at checkpoints or when blocked, and are commonly given stopping conditions such as a maximum number of iterations. In a plan revision test, the handoff note is where that ground truth either survives or disappears.
Checklist · 6 items
- Active constraints, with the step where each was introduced.
- Authority currently held, who granted it and whether the trigger changed it.
- Commitments made to people outside the system, such as a promised time window.
- Open conflicts between sources and which one the plan currently relies on.
- Steps already executed that cannot be undone.
- The evidence that caused the latest revision, by pointer, so the receiver can check it.
0 of 6 ticked. Tick each item as you go.
Illustrative: a dispatch agent learns a part was substituted
This case is invented to show the annotation; the counts are illustrative. A dispatch agent plans a 24-step job: confirm the customer's time window, reserve a part, assign a technician with the right certification, book the visit, send the confirmation. At step 9 the warehouse lookup returns a substituted part that requires a different certification.
The gold annotation marks 3 steps invalidated (technician assignment, booking and the confirmation draft) and 2 commitments preserved (the customer's time window and a note that the site requires an escort). Expected response: local repair. The control twin returns the original part at step 9.
One run detects the change at step 10, reassigns the technician and keeps the time window, but rewrites the confirmation without the escort note: recall 3 of 3, preservation 1 of 2. A second run never cites the substitution and books the original technician: detection blank, revision recall 0 of 3. A third run handles the trigger correctly but, in the control twin, rebooks the customer anyway: it fails restraint. Each failure points to a different fix, which a single outcome score would not show.
When revising the plan is the wrong move
Not every new piece of evidence deserves a new plan. Revising is wrong when the evidence comes from an untrusted channel, when it conflicts with a more authoritative source or when acting on it would exceed the agent's authority. In those cases the correct response is to verify, flag or escalate, and the annotation should say so.
Tool output and inbound messages can also carry instructions that only look like evidence. Include at least one trigger where text inside a document or email tells the agent to change its plan, and mark expected_response as none or escalate, so the test rewards an agent that treats content as data.
Stop and bring in a domain expert before finalizing gold labels when the correct response depends on policy the annotators do not own, such as refund rules, safety procedures or approval limits. Measure agreement between two annotators on invalidated_steps and expected_response; where they disagree often, the task is underspecified and should be rewritten rather than scored.
Finding natural plan changes in operational records
Synthetic triggers prove the scoring works; real operational history shows how plans actually change. Change orders in project tools, reassigned and reopened tickets in help desks, rescheduled jobs in field service systems and amended purchase orders in ERPs each record a plan, the evidence that changed it and what happened next. Records like these, with timestamps and authorship intact, make triggers and gold revisions far more realistic than invented ones.
Using them requires more than access. Holding system access, having privacy permission to process the records, owning the business records and having a contractual right to license them are separate questions.
For agent teams that want this kind of change history, SourceX works request by request: it searches among U.S. businesses for the records described, rather than offering a ready-made catalog, and the request captures scope, volume, language, format, timing and licensing needs. A dispatch or project history reaches a buyer only after its owner has approved the scope and terms and signed.
What should a dataset request for plan-revision records ask suppliers to deliver?
Ask for linked change histories that fill the annotation fields above from real work: the state before a change, the evidence that changed it, what was revised and what was kept, who had authority, the reason written at the time and the outcome. Finished plans alone show what a good plan looks like, not when or why one should change.
Write the request from a failure you can already reproduce with the annotation guide. If agents keep acting on revoked approvals, make authority changes the required trigger type and ask for the revocation, the response and what followed; if they rebuild plans around irrelevant messages, ask for events that left the plan untouched. Run the acceptance column on the supplier's sample before licensing the rest.
| Annotation field it feeds | What the records must contain | Acceptance test on the sample | What is lost without it |
|---|---|---|---|
| plan_before | The plan, schedule or ticket state before the change, with step or task identifiers and a timestamp | Rebuild the state as of the trigger time for 20 drawn cases using only fields that existed then | No baseline, so no revision to score |
| trigger_step and evidence_pointer | The changed field, message or record, with old value, new value, time and source system | Every drawn case shows old and new values, not only the final state | Triggers have to be invented |
| invalidated_steps | The tasks, dates or assignments changed after the trigger | Each later change links back to the trigger by ticket, order or change identifier | Revision scope cannot be measured |
| preserved_commitments | Promises and constraints recorded before the trigger, such as due dates, customer windows, approvals and site rules | Each commitment carries a creation time earlier than the trigger | Preservation cannot be scored |
| expected_response | Who requested and approved the change, and the role or limit that allowed it | Approver roles are present and pseudonymized consistently | Escalation and local repair look the same |
| Revision reason | The reason, comment or approval note written when the change was made | Reason timestamps fall at or before the revision; none were added during export | Reasons reflect hindsight |
| End state | Final status and dates after the revision, such as fulfilled, missed or reopened | Most drawn cases reach an outcome inside the export window | A revision cannot be tied to success |
| Control twins | Changes and messages that arrived but left the plan untouched, drawn by the same rule | Present at the share stated in the request | No real-data test of restraint |
Which systems record the change, the reason and who approved it?
Systems that log field-level changes with time, author and previous value can supply most of the brief, but only for fields that were being logged during the period you want. Confirm what was recorded before agreeing scope, because history that was never logged cannot be exported later.
Three documented examples show the range. Jira's REST API returns a paginated list of all changelogs for an issue, sorted by date from the oldest. Zendesk describes ticket audits as a read-only history of a ticket's updates, each able to hold several events, and its change events carry the previous value. Microsoft's Dataverse auditing records who changed a record, when, which fields and their earlier values, but only where an administrator turned auditing on, and it deletes logs older than the configured retention period unless retention is set to forever.
These logs record what changed far more reliably than why. A reason field that was optional, or a comment written days later, fails the reason row of the brief, so ask for the share of drawn changes that carry a reason timestamped at or before the revision before pricing the full set. The list maps the trigger taxonomy above to the records that tend to carry each trigger.
- Tool result contradicts an assumption: availability or status fields that change on an order or work order, with the old value kept in the history.
- New requirement from a person: due-date or priority edits linked to the customer comment that asked for them.
- Authority change: approval status edits with the approver and time, including revocations.
- Shared state changed by another actor: assignee or owner changes in an issue or ticket history.
- Time passes: service-level or due-date breaches recorded as events when they happened, not inferred later.
- Irrelevant new evidence: comments and notifications on the same record that were followed by no change to plan fields.
When will plan-revision records not fix the agent?
When the agent never saw the change, could not reach the system holding it, or works in a process that never records the reason, more records will not help. Run the four checks below on the failing runs before writing the request.
Only failures that survive all four belong in the request, each with its trigger type, the step where it appeared and the end state you expect, so the supplier's sample can be scored with the same fields. No validated plan-revision study accompanies this guide; the brief is a method, and the first sample is where it gets tested. When the request goes through SourceX, which sources records on request rather than from stock, the trigger types and acceptance tests belong in the request's scope.
- Observation: replay the failing runs and confirm the change appeared in a tool result the agent actually received. A cached or truncated response is a tooling fix.
- Reach: confirm the agent had a tool and permission to query the system where the change lived. A revocation stored where the agent cannot read is an access gap.
- Process: confirm the business records the change and its reason at all. Plans revised in meetings and never logged leave no trigger for an agent or a dataset to learn from.
- Grading: confirm the expected revision is unambiguous. Anthropic's engineering team warns that ambiguity in task specifications becomes noise in metrics, and says reading transcripts shows whether a failure was a genuine agent mistake.
Sources for this section
- Jira Cloud's REST API returns a paginated list of all changelogs for an issue, sorted by date and starting from the oldest. Source
- Zendesk describes ticket audits as a read-only history of a ticket's updates, where each audit can contain several events. Source
- Zendesk's ticket audit events reference lists change events, which include the previous value of the changed field. Source
- Microsoft's Dataverse auditing records who changed a record, when, which fields and their previous values, is turned on by an administrator, and deletes logs older than the retention period unless retention is set to forever. Source
- Anthropic's engineering post (January 9, 2026) says ambiguity in task specifications becomes noise in metrics and that transcripts show whether a failed task was a genuine agent mistake. Source
Frequently asked questions
What is an evidence trigger in agent evaluation?
It is a planned change in what the agent can observe, such as a tool result, a message or a record update, placed at a known step. Because the step and the change are known in advance, annotators can define the correct revision before any agent runs.
Why include tasks where nothing should change?
Without them, an agent that rewrites its plan at every new message looks responsive. Control twins measure restraint, which is as important as detection in long workflows where many inputs are irrelevant.
Is final task success enough to evaluate replanning?
No. End state checks remain essential, but an agent can succeed without revising or fail for unrelated reasons. Label detection, revision scope, preservation and response type separately, then report end state and repeated-run reliability alongside them.
How many annotators does a plan revision task need?
At least two for the gold fields, especially invalidated_steps and expected_response. Disagreement is a signal that the task or its policy is unclear and should be fixed before scoring agents.
Can real business records be used to build these tests?
Yes, when rights are in place. Change orders, reassigned tickets and rescheduled jobs show natural triggers and revisions. Confirm ownership, privacy permission and a license to use them for AI before they enter a test set.
Sources
- ReAct (Yao et al.) interleaves reasoning traces with actions and argues the reasoning helps the model induce, track and update action plans and handle exceptions. Source
- tau-bench evaluates tool-using agents by comparing the final database state with the annotated goal state and introduces pass^k, the probability that all k trials of a task succeed. Source
- The tau-bench repository reports Pass^1 through Pass^4 for each agent strategy and directs users to its successor benchmark for updated tasks. Source
- TravelPlanner evaluates planning agents on hard and commonsense constraints and reports a hard constraint pass rate and a final pass rate. Source
- METR's paper measures AI agent capability as a time horizon defined by how long tasks take human professionals. Source
- Reflexion agents verbally reflect on task feedback and keep reflective text in an episodic memory buffer to improve later trials. Source
- Anthropic's guide says agents need ground truth from the environment at each step, can pause for human feedback at checkpoints, and often run with stopping conditions such as a maximum number of iterations. Source
- AgentBoard evaluates agents in multi-round, partially observable environments and reports fine-grained progress rates. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.