AI uses for records
Agent memory stale facts benchmark: will your agent notice a fact went out of date?
By SourceX Editorial · Updated
10 sections
Short answer
Test it with a dated record stream: teach the agent a fact with its source and valid-from date, later supply an authoritative, dated change, then ask questions and give tasks whose right answer depends on noticing it. Score whether the agent uses the current value, answers past-date questions with the old one, ignores lower-authority claims, respects withdrawn permission and never resurfaces deleted facts.
Agent run
Agent memory stale facts
Key takeaways
- A stale fact is a value the agent still treats as current after a more authoritative, dated record replaced, corrected, expired or revoked it.
- Keep valid time and record time separately in the gold data, so missing a change can be told apart from never being told.
- Cover seven change types, including lower-authority claims that should not trigger an update and deletions that must never resurface.
- Use probes that rely on the fact inside a task, not only direct questions, because recall and noticing fail differently.
- Report each metric by change type and distance with intervals, and version the benchmark because its gold answers age too.
What does it mean for a remembered fact to go stale?
A remembered fact is stale when the agent still treats a value as current after a more authoritative, dated record has replaced, corrected, expired or revoked it. The agent may have stored the value correctly at the time; what failed is the link between the memory and the moment it stopped being true.
Two clocks matter. Valid time is when the fact was true in the world, such as the date a customer's billing contact changed. Record time is when the agent or a system learned about it. A benchmark that stores only one timestamp cannot tell an agent that missed a change from one that was never told about it.
The Graphiti project, an open-source framework for temporal knowledge graphs, describes each fact as carrying a validity window: when it became true and when, if ever, it was superseded, with old facts invalidated rather than deleted so that past states stay queryable. Wikidata uses the same idea with start time and end time qualifiers on a statement. Both are useful models for the gold data in a stale-fact test, even if the agent under test stores memory differently.
Staleness is also different from error. A correction says the old value was never right; a supersession says it was right until a date. An agent that answers a historical question with the corrected value is as wrong as one that answers a current question with the superseded value, so the benchmark has to label which kind of change happened.
Why do public memory benchmarks only partly test staleness?
Public memory benchmarks test whether a chat assistant tracks updates across long conversations, which is a necessary skill but not the whole problem for an enterprise agent. They rarely model source authority, permissions or the gap between a change event and the record that reports it.
LongMemEval releases 500 questions covering five long-term memory abilities, one of which is knowledge updates, alongside information extraction, multi-session reasoning, temporal reasoning and abstention. Its chat histories are timestamped, and turns holding the needed evidence are flagged so that recall can be scored at the turn and session level. LoCoMo provides very long conversations split into dated sessions, with question annotations that point to the dialog turns containing the answer.
FreshQA comes at the problem from world knowledge. The FreshLLMs paper sorts questions into never-changing, slow-changing and fast-changing answers, plus false-premise questions the model should rebut, and notes that a question can move between categories as the world changes. The maintainers say they update the dataset weekly or on request.
What these leave out is the shape of business change. In a company, a fact changes because a system of record changed, an authorized person said so, a contract term ended or a permission was withdrawn. A user remark in chat may be right, wrong or out of scope. The benchmark below borrows the dated-evidence and abstention ideas from the public sets and adds authority, permission and correction history.
Which kinds of change should the test cover?
Cover at least the seven change types below, because each one calls for a different correct behavior and fails in a different way. A test set built only from simple overwrites will reward an agent that updates on any new text, which is the wrong habit for business records.
| Change type | Business example | Correct agent behavior | Failure to score |
|---|---|---|---|
| Supersession | The billing contact on an account is replaced in the CRM on a dated edit | Use the new value for current questions; keep the old value for questions about the earlier period | Answers with the old contact after the change date |
| Correction | A data steward fixes a mistyped contract start date | Replace the value for all periods and note that the earlier value was an error | Treats the wrong value as having been true until the fix |
| Scheduled expiry | A discount or service tier ends on a date written in the original record | Stop using the value after the end date without being told again | Keeps applying the term after it lapsed |
| Permission withdrawn | A user revokes consent to use a stored preference | Stop using the fact for the withdrawn purpose even though it is still accurate | Keeps personalizing with the revoked fact |
| Deletion request | A record owner asks for a remembered personal detail to be removed | Remove it from memory and derived summaries, then confirm nothing resurfaces | The detail returns in a later summary or answer |
| Lower-authority claim | A caller says the account manager changed, but the CRM still shows the old one | Flag the conflict, keep the system value and ask for confirmation | Overwrites the system value on an unverified remark |
| False premise | A question assumes a contract renewed when the record shows it was cancelled | Rebut the premise with the dated record | Answers as though the premise were true |
What should the stale fact benchmark with dated evidence include?
Each benchmark item is one fact with a dated life story: where it came from, who had authority over it, what changed it, when, and what the agent is allowed to do with it. The table lists the fields; the probes and expected answers in the next section hang off them. Keep every field even when a value is empty, because an explicit null is evidence that the gap was checked.
At a glance
What should the stale fact benchmark with dated evidence include
Store the evidence, not just the label. For every value and every change, keep a pointer to the source record and its own timestamp, so a reviewer can confirm the gold answer without trusting the item writer. This is the dated-evidence part of the benchmark, and it is what lets a buyer audit the test later.
| Field | What it holds | Why the benchmark needs it |
|---|---|---|
| fact_id and subject | Stable id plus the entity and attribute, such as account 4471 billing contact | Lets probes, changes and deletions refer to the same fact |
| value_v1 and source_v1 | The first value and the record that stated it, with system name and record id | Proves the agent was taught a value that was correct at the time |
| authority_rank | Ordered list of sources allowed to change this attribute, such as CRM over email over chat | Decides whether a later claim should update memory or only raise a flag |
| valid_from and recorded_at | When the value became true and when the agent first saw it | Separates missing a change from never being told |
| change_event | Type from the seven-type list, event date, source record, authority and actor role | Defines what the agent should have noticed and from where |
| value_v2 and valid_to | New value, or null for revocation and deletion, and the end of the old window | Gold answer for current and as-of questions |
| permission_scope | Purposes the fact may be used for and any withdrawal date | Tests use limits separately from accuracy |
| probe_set | Questions and tasks at stated distances after the change, each with its expected answer | Measures noticing, not just recall |
| expected_memory_action | Update, invalidate and keep history, forget, flag conflict or no change | Scores the write decision, not only the answer |
| review_record | Who checked the gold answer against the source records, and on what date | Makes the item auditable and dates its validity |
How do you write probes that separate noticing from guessing?
Ask the agent questions and give it tasks whose correct output depends on the change, at several distances after the change, and pair each with a control item where nothing changed. Without the control, an agent that always prefers the newest text looks as good as one that reasons about authority.
Distance matters because many failures appear only when the change is buried. Place the change event at varied positions: immediately before the probe, several sessions earlier, and behind distractor sessions that mention the old value again in passing. Record the distance in sessions and elapsed days on every probe so results can be broken out by it.
Include tasks where the fact is used rather than asked about. Drafting an invoice email that should go to the new billing contact tests noticing better than asking who the billing contact is, because the agent has to retrieve the fact without being prompted.
- Current-value probe: what is the value now, answered from the latest authoritative record.
- As-of probe: what was the value on a stated date before the change, which the old value should answer.
- History probe: when did it change and what was it before, which tests whether the agent kept the old window.
- Use probe: a task that silently depends on the fact, such as routing, addressing or applying a term.
- False-premise probe: a question that assumes the stale value, which the agent should correct.
- Conflict probe: a lower-authority remark contradicts the record, and the agent should flag rather than switch.
- Deletion probe: a later summary or question where a removed fact must not appear.
- Control probe: the same question on a matched fact that never changed.
How should you score update, forget and abstain decisions?
Score the memory decision and the answer separately, and report both by change type and distance rather than as one blended accuracy. A single number hides the most expensive failures, such as an agent that handles supersession well but resurfaces deleted details.
Watch the denominators. The over-update rate is computed only on items whose expected action is to flag or keep, and the stale-answer rate only on items with a real change; mixing them into one pool lets a cautious agent and a credulous one post the same score for opposite reasons.
Run every item several times when the agent is not deterministic and report the share of trials that pass, not only whether any trial passed. Report counts and confidence intervals per cell. Many cells will be small, and a wide interval is a finding in itself: it tells the reader which change types need more items before anyone relies on the score.
| Metric | Counts as a pass when | Reported by |
|---|---|---|
| Stale-answer rate | Current and use probes return the post-change value or a correct abstention | Change type and distance |
| History accuracy | As-of and history probes return the value valid on the stated date | Change type |
| Over-update rate | The agent keeps the system value when only a lower-authority claim changed | Source pair, such as chat versus CRM |
| Premise rebuttal | The agent corrects a stale premise and cites the dated record | Distance |
| Permission respect | A revoked fact is not used for the withdrawn purpose | Purpose |
| Deletion leakage | A deleted fact appears in no later answer, summary or tool argument | Distance and output channel |
| Abstention precision | The agent abstains only when authority, date or permission is genuinely unclear | Item ambiguity label |
Illustrative worked example: a billing contact changes in the CRM
This example is illustrative; the systems, counts and timings are invented to show the method, and no results are reported. An account in a CRM lists a billing contact. On day 0 the agent reads the record and stores the contact. On day 12 a finance administrator edits the CRM field to a new person, and the edit is the change event, with the CRM as the top authority for this attribute.
Between day 12 and day 30 the item feeds the agent five unrelated sessions, two of which mention the old contact in passing in quoted email threads. On day 30 a support chat user writes that the account manager has also changed, which the CRM does not show; that is a conflict probe, and the expected action is to flag it, not to update.
The probes on day 30 are: who is the billing contact now (new person); who was it on day 5 (old person); draft the overdue-invoice email (addressed to the new person); a premise question asking whether the old contact approved last month's invoice, which the agent should answer from dated records rather than assume; and a control question on a second account whose contact never changed.
Reading the illustrative scorecard, an agent that answers the current question correctly but addresses the invoice email to the old contact has passed recall and failed noticing. That pattern is the main reason to include use probes.
When should the agent keep the old value instead of updating?
Keep the old value when the new information comes from a source without authority over that attribute, when the question is about an earlier period, or when the record is under a hold that freezes it. Updating in those cases is a failure, and the benchmark should score it as one.
Lower-authority conflicts are the most common case. A customer's remark that their address changed is useful, but in many workflows it should trigger verification, not a silent overwrite of the system of record. Label these items with the expected action of flagging the conflict and asking the right owner.
Historical questions are the second case. Audit, dispute and reconciliation work need what was true on a date, and an agent that only keeps the latest value cannot answer them. This is why the gold data keeps validity windows instead of overwriting.
The third case is out of scope for most test sets: values frozen by a legal or contractual hold. If your agent operates where holds apply, add items for them and have the records owner write the expected behavior, since the right answer depends on company policy and, where relevant, on advice from counsel.
How do you keep the benchmark itself from going stale?
Date every gold answer and give the benchmark a version, because a test about changing facts decays like the facts it tests. FreshQA's maintainers handle this by updating answers on a published schedule and keeping older versions available; a private stale-fact set needs the same discipline.
Write a review date and a refresh trigger into each item. When a source system, an authority rule or a permission policy changes, the items that depend on it should be rechecked or retired, and the release notes should say which ones moved. Scores from different versions should not be compared without a note on what changed.
Know the limits of the method. It measures behavior on the change types you wrote, at the distances you tested, with the memory architecture you ran. It does not show how the agent behaves on attributes you did not include, on far longer histories, or when the source systems themselves are wrong.
Where can real change histories for a stale-fact test set come from?
The most valuable raw material for this benchmark is a real change history: field edits with timestamps and actor roles, ticket audit trails, contract amendments and consent changes, exported with the records they modify. Synthetic stories can fill gaps, but they tend to miss the messy cases, such as corrections made twice or changes recorded days after they took effect.
Because SourceX finds change histories at U.S. businesses only when a buyer asks, with nothing kept in stock, the request itself sets the scope, volume, format, timing and licensing terms, for example dated field histories from a CRM or service desk. Records reach the buyer only after the company holding them has signed off on scope and terms and executed the agreement.
Frequently asked questions
Is a knowledge-update test the same as a stale-fact benchmark?
No. A knowledge-update test, like the category in LongMemEval, checks whether the agent uses the latest value stated in a conversation. A stale-fact benchmark also checks source authority, as-of questions, permission withdrawal, deletion and conflicts that should not cause an update.
How many items does a stale-fact benchmark need?
Enough per change type and distance that the confidence interval in each cell is narrow enough for the decision you are making. Start by counting the cells you intend to report, then size each one; a set that is large overall but thin per change type cannot tell you where the agent fails.
Should an agent delete an outdated fact or keep it?
Usually it should invalidate the fact and keep it as history with its validity window, so it can still answer questions about the past. Deletion is a different event: when a record owner asks for removal, the fact should leave memory and derived summaries, and the test should confirm it never resurfaces.
Can synthetic conversations stand in for real change histories?
They can cover rare change types and controlled distances, but they tend to be cleaner than real records. Mix synthetic items with items built from real dated audit trails and report the two groups separately so you can see whether scores hold on real data.
How do you test that a deleted memory does not come back?
Delete the fact at a known point, then probe it later through direct questions, summaries and tasks whose tool arguments could carry it. Count any appearance in any output channel as leakage, and report it by distance after the deletion.
Sources
- LongMemEval releases 500 questions testing five long-term memory abilities, including knowledge updates, temporal reasoning and abstention, over timestamped chat histories with evidence turns flagged. Source
- LoCoMo provides very long conversations split into dated sessions, with question annotations listing the dialog turns that contain the answer. Source
- Graphiti describes facts with validity windows, invalidating rather than deleting old facts so earlier states remain queryable, and calls this bi-temporal tracking. Source
- Wikidata uses start time and end time qualifiers to state the period during which a statement was true. Source
- The FreshLLMs paper sorts FreshQA questions into never-changing, slow-changing, fast-changing and false-premise questions and notes that questions can be reclassified over time. Source
- The FreshQA maintainers say they update the dataset weekly or on request and keep earlier versions available. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.