AI uses for records
Long horizon agent evidence refresh: testing when a source changes
By SourceX Editorial · Updated
10 sections
Short answer
To test an agent when a source changes during its task, edit the source on purpose between the agent's read and its use, then check whether it noticed, re-read before acting, and left the records matching the current version. Run twins with no change and with an irrelevant change, plant changes inside waits and handoffs, and score stale use, needless rework and end state across repeated trials.
Agent run
Long horizon agent evidence
Key takeaways
- Plant changes between the agent's read and its use, especially inside waits, approvals and handoffs.
- Pair every changed run with a no-change twin and an irrelevant-change twin to separate vigilance from habit.
- Expose the change signals real systems use, such as ETags, If-Match, resourceVersion and delta feeds.
- Score stale use and needless rework together, by change type and signal condition, never as one average.
- Include point-in-time tasks where ignoring a newer version is the correct answer.
How do you test an agent when a source changes during its task?
Change the source on purpose at a known moment between when the agent reads it and when the agent relies on it, then check three things: whether the agent noticed, whether it re-read before acting, and whether its final output and the records it changed reflect the current version. Pair every changed run with a twin where the source stays put or changes in a field that does not matter, so you can tell vigilance from habit.
Long-horizon agents make this unavoidable. A task that spans hours of tool calls, waits for approvals or passes between agents will read a spreadsheet, a ticket, a policy page or an inventory record early and use it much later. In between, a person edits it, a sync job overwrites it or it is deleted. An agent that treats its first read as permanent will act on a world that no longer exists.
The freshness and source revalidation benchmark design in this guide turns that risk into a repeatable test. It specifies which sources change, when, how the change is exposed, what the agent should do, and how each behavior is scored.
Which kinds of source change should the benchmark plant?
Plant at least four kinds, because each one breaks a different assumption. A benchmark that only edits values will miss agents that cannot cope with a deleted record or a document that was replaced by a newer authority.
| Change type | Example in a business system | Assumption it breaks | Correct agent response |
|---|---|---|---|
| Value edit | A technician's availability cell changes in a scheduling sheet | What I read is still true | Re-read before committing anything that depends on the value |
| Append | A new comment on a ticket adds a customer constraint | I saw the whole record | Re-read the full thread before closing or replying |
| Removal or expiry | A purchase order is canceled or a record is purged | The thing I am acting on still exists | Stop dependent steps and report the missing source |
| Supersession | A pricing or returns policy page is replaced by a new version | The document I cited is the one in force | Switch to the new version and say which version the output follows |
| Irrelevant change (control) | An unrelated column or a formatting edit | Every change matters | Notice if cheap, but do not redo work or alter the result |
What signals do real systems give that a source has changed?
Most production APIs give a version token, a timestamp or a change feed, and the benchmark should expose the same ones so the agent is tested on the signals it will meet. The point is to reward an agent that uses these cheap checks instead of either re-reading everything or trusting nothing.
HTTP itself defines the vocabulary. RFC 9111 says a response is fresh while its age is below its freshness lifetime and stale after that, which gives a benchmark a precise way to state how old a read may be. RFC 9110 defines conditional requests: a client sends If-Match with the version it read, and if the condition is false the server must not perform the change and may answer 412 Precondition Failed, which the RFC presents as the guard against the lost update problem.
Vendor APIs follow the same pattern. GitHub's REST documentation says most endpoints return an etag and many a last-modified header, and that a conditional GET returns 304 Not Modified when nothing changed, without counting against the primary rate limit when properly authorized. The Kubernetes API concepts page says a PUT must carry the object's resourceVersion and that a stale one gets a 409 Conflict, and that a server may answer 410 Gone when asked for a version older than it keeps. Microsoft Graph's delta query returns a deltaLink that reports changes since the previous request, with one catch worth testing: if the client selected only some properties, changes to the others do not appear.
| Mechanism | How a change shows up | What the agent should do | Failure the benchmark can plant |
|---|---|---|---|
| Freshness lifetime (RFC 9111) | The stored response's age exceeds its lifetime | Revalidate before relying on it | A cached read used after it went stale |
| ETag and conditional GET | A 200 with a new etag instead of 304 | Replace its working copy and recheck dependent steps | The agent ignores the new etag |
| If-Match on a write (RFC 9110) | 412 Precondition Failed | Re-read, reconcile, then decide whether to retry | The agent retries the write without the precondition |
| resourceVersion (Kubernetes) | 409 Conflict on update, 410 Gone on an old version | Fetch the current object and re-plan the update | The agent overwrites with stale fields |
| Delta feed (Microsoft Graph) | Changed items arrive on the next deltaLink call | Merge changes and check them against the plan | A change to an unselected property never arrives |
When must an agent re-read a source it has already used?
Before any step that is hard to undo, after any wait or handoff, and before it reports a final answer that depends on the source. Between those points, re-reading every few steps mostly costs time and quota without changing the result.
The useful way to think about it is that each piece of evidence has a read time and a use time. The gap between them is the window in which the world can drift. A short gap before reading and writing a value in the same breath is low risk. A gap that spans an approval queue, an overnight pause or a transfer to another agent is high risk, and the benchmark should put its planted changes inside exactly those gaps.
The agent should also carry the version it read along with the content. A plan that says the technician is free on Thursday is far less useful to the next step than one that says so as of a specific etag or modified time, because only the second can be checked cheaply.
The freshness and source revalidation benchmark design
Each task in the benchmark is defined by the components below. Version the whole design and publish it with the date, so results from different versions are never pooled. This is a design for running the test; it reports no results.
At a glance
The freshness and source revalidation benchmark design
| Component | What to specify | Why it matters | Recorded per run |
|---|---|---|---|
| Task and goal state | The business outcome, written as records that must exist at the end | Grading needs a target that is not the agent's own summary | Goal state per condition |
| Source inventory | Every source the agent may read, its owner and who is allowed to change it | Authority decides whether a change should be followed | Source ids, owners, permission to edit |
| Dependency map | Which steps and outputs depend on which source fields | Separates relevant changes from noise | Field-to-step links |
| Mutation schedule | Which source changes, which field, and at which point relative to read and use | Places changes in the read-use window | Mutation time, step index, old and new values |
| Exposure condition | Whether a validator, a change feed or nothing at all is available | Tests both signal use and blind rechecking | Signals offered to the agent |
| Control twins | The same task with no change and with an irrelevant change | Measures over-reaction and needless rework | Twin pairing id |
| Handoff points | Where the task pauses or passes to another agent or a person | Changes during handoffs are the hardest to catch | Handoff packet contents |
| Expected behavior | Re-read, reconcile, stop, or ignore, for each mutation | Lets graders judge process, not only outcome | Expected action per mutation |
| Trials | How many repeated runs per task and condition | Timing makes outcomes vary | Run index and seed |
How should each run be scored?
Score the end state first and the process second, and report every metric by change type and exposure condition rather than as one blended number. An agent that does well on value edits with an etag and badly on supersession with no signal has a specific weakness that an average would hide.
End-state grading follows the approach of the tau-bench paper, which compared the database state at the end of each conversation with an annotated goal state and introduced pass^k to measure whether an agent succeeds across all of k independent trials. Freshness failures are often timing dependent, so a pass in every trial is the bar worth reporting.
| Metric | Definition | Read with |
|---|---|---|
| Stale use | Share of relevant mutations after which the agent acted on the old value | Change type and exposure condition |
| Revalidation at risk points | Share of irreversible steps, handoffs and final answers preceded by a fresh read or a conditional check | Number of risk points per task |
| Needless rework | Steps redone or outputs altered in irrelevant-change and no-change twins | Stale use, since both should be low |
| Version citation | Share of outputs that name the version or time of the source they rely on | Whether the source exposed a version |
| Conflict handling | Correct response to 412, 409 or 410 style rejections | Number of planted conflicts |
| End state and pass^k | Goal state reached in every one of k trials | All of the above |
What must a handoff packet carry so the next agent can recheck a source?
The version token is the item most often lost. A handoff note that says the order quantity is twelve, without the record's version or read time, invites the next agent to act on it without checking. Require the packet to carry, for each source it relies on, the content used, the version or modified time, when it was read and the dependent steps still open.
Authority gets lost next. If a human editor changed the source during the pause, the next agent needs to know whether that person was allowed to change it and whether the change overrides the original instruction. The source inventory in the design records who may edit what, so the grader can check that the agent followed an authorized change and flagged an unauthorized one rather than silently adopting it.
Permissions can also drift. A token scope or a shared folder setting that changed during a long wait turns a freshness problem into an access problem, and the agent should report it as such instead of retrying.
Illustrative worked task: a field schedule built from a sheet that moves
This example is illustrative; the counts and step numbers are invented to show the design. An agent builds next week's field service schedule for 6 technicians and 30 jobs from an availability sheet and a job list, then writes the bookings to a scheduling system after a manager approves the draft.
The mutation schedule plants one relevant change: after the agent reads the sheet at step 4 but before the bookings are written at step 19, one technician's Thursday is marked unavailable. Two twins run the same task, one with no change and one in which only a notes column is edited. The sheet exposes a modified time and the scheduling API supports conditional writes.
A passing run re-reads the sheet after the approval pause, sees the new modified time, moves the affected Thursday jobs, tells the manager what changed since approval and writes bookings that match the current sheet in every trial. Common failures are booking the unavailable technician, re-running the whole schedule in the notes-only twin, or moving jobs without telling the approver that the plan changed after sign-off.
- Goal state, changed condition: no Thursday bookings for the unavailable technician; all 30 jobs booked once.
- Goal state, no-change twin: bookings identical to the approved draft.
- Goal state, irrelevant-change twin: bookings identical to the approved draft, no re-approval request.
- Process expectation: a fresh read or conditional check between the approval and the first booking write.
When should an agent deliberately ignore a newer version?
When the task is defined at a point in time. A month-end report, an audit sample or a reply that must reflect the policy a customer agreed to should be built on a fixed snapshot, and an agent that silently pulls in later edits is wrong. The benchmark needs these snapshot tasks too, with the as-of time stated in the task, so revalidation is scored as a judgment rather than a reflex.
Knowledge freshness is a related but different test. FreshQA, from the FreshLLMs paper, groups questions as never-changing, slow-changing, fast-changing or false-premise, and its repository says the dataset is updated weekly or upon request. That measures whether a model's answers about the world are current; this design measures whether an agent notices that a specific source changed while it was working.
Some sources expose no validator at all. The design handles them with the no-signal exposure condition, but results there say more about the agent's rereading habits than about its use of change signals, and they should be reported separately.
Where real source change histories come from
Synthetic mutations prove the harness works; real edit histories make the changes realistic. Revision histories in shared spreadsheets and documents, ticket audit trails, purchase order amendments and policy page versions all record what changed, when and by whom, which is exactly what a mutation schedule needs.
When a lab asks, SourceX searches U.S. businesses for edit histories like these rather than selling from a catalog, and the request captures scope, volume, language, format, timing and licensing needs. The business holding the histories signs the agreement, with scope and terms approved, before anything is delivered.
Frequently asked questions
What is evidence refresh for a long-horizon agent?
It is the agent's practice of re-reading or revalidating a source it used earlier before relying on it again. It matters most before irreversible actions, after waits or handoffs, and before a final answer, because those are the points where the source has had time to change.
How is this different from testing plan revision?
A plan revision test asks whether the agent changes its plan when new evidence arrives. A freshness test asks an earlier question: whether the agent notices that a source it already read is out of date at all, and whether it checks at the right moments.
Should an agent re-read every source before every step?
No. That wastes time and API quota and rarely changes the outcome. A good agent uses version tokens or modified times where they exist and re-reads at risk points, which the benchmark rewards through the needless rework metric.
What if the source has no ETag or version number?
Run those tasks under a separate no-signal condition and report them apart from the rest. The agent can still re-read and compare content, but the result reflects its rereading habits rather than its use of change signals.
How many trials does each task need?
Several per task and condition, because the timing of a planted change relative to the agent's reads varies between runs. Report success in every trial, for example with pass^k, rather than the best run.
Sources
- RFC 9111 defines a response as fresh while its age is below its freshness lifetime and stale otherwise. Source
- RFC 9110 says a server must not perform a method when an If-Match condition is false, may answer 412 Precondition Failed, and presents conditional writes as the guard against lost updates. Source
- GitHub's REST documentation describes etag and last-modified headers and conditional GET requests that return 304 Not Modified without using the primary rate limit when authorized. Source
- The Kubernetes API concepts page says an update with a stale resourceVersion gets 409 Conflict and that servers may return 410 Gone for versions older than they retain. Source
- Microsoft Graph's delta query returns changes since the previous request through a deltaLink, and changes to properties outside a $select do not appear. Source
- The tau-bench paper grades agents by comparing the final database state with an annotated goal state and introduces pass^k for reliability across trials. Source
- The FreshLLMs paper introduces FreshQA with never-changing, slow-changing, fast-changing and false-premise questions. Source
- The FreshQA repository says the dataset is updated weekly or upon request. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.