AI uses for records
Agent memory evaluation dataset: what to keep after first contact
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
After a first customer conversation, an agent should remember only facts it can tie to a speaker, a time and a permission, that the speaker had authority to state, and that stay true until something changes them. An agent memory evaluation dataset tests this with labeled items (retain, retain until a date, supersede, forget on request, never store, abstain, escalate a conflict) and probes them in later sessions.
Agent run
Agent memory evaluation
Key takeaways
- Treat every remembered fact as a dated claim with a speaker, an authority and a permission, not as a truth.
- Keep account facts, stated preferences, dated commitments and open issues; skip speculation, third-party details and unneeded sensitive data.
- Give each candidate memory one of seven handling labels before any model sees it, and grade each label separately.
- Place probes after the correction, expiry or deletion they test, and require a pointer to the source turn.
- Deletion tests must cover every store, because memory and chat history are often kept apart.
- A request not to be contacted is the exception: it must survive a general forget request.
What should an agent remember after a first customer conversation?
Keep what the customer will expect you to know next time, and nothing that you would struggle to defend if they asked where it came from. In practice that means four kinds of facts: stable account facts (who the account is, which product or site is involved), stated preferences (channel, timing, language), commitments with dates (a callback promised for Thursday, a renewal decision expected next quarter) and open issues with their current state.
Leave out guesses the agent made about the customer, one-off context that will not matter again, details about third parties the speaker had no standing to share, and sensitive details that the task did not need. A first conversation is noisy: people think aloud, correct themselves and mention colleagues. An agent that stores every sentence will later act on half-formed statements as if they were settled.
The test for an agent memory evaluation dataset is therefore not recall alone. It asks whether the agent kept the right facts, attached the right limits to them, changed them when the customer corrected them, dropped them when asked, and admitted it did not know when the fact was never said.
Why is a remembered fact a claim and not a truth?
A memory item is a statement someone made at a particular time, with a particular authority, under a particular permission, and it can become false. Treating it as a dated claim, rather than a fact about the world, is what lets you test correction, expiry and deletion at all.
Temporal memory frameworks already model this. Graphiti, an open-source framework for temporal knowledge graphs for agents, describes each fact as having a validity window, when it became true and when, if ever, it was superseded, and traces facts back to the episodes that produced them. Its edge model carries separate valid_at and invalid_at timestamps, and its contradiction handling invalidates an old fact while keeping its history rather than overwriting it.
Three times matter for every item: when the conversation happened (record time), when the fact is said to hold (valid time, which may start in the future, as with a planned move), and when the agent last confirmed it. Collapsing them into one timestamp is the most common reason a memory test cannot tell a stale answer from a correct one.
What should the memory evaluation set include?
Each row in the set is one candidate memory extracted from a real or carefully constructed first conversation, plus everything needed to grade what the agent does with it later. The fields below are the minimum that let a grader separate a correct update from a lucky guess. Example values are illustrative.
At a glance
What should the memory evaluation set include
| Field | What it records | Illustrative value | Why the grader needs it |
|---|---|---|---|
| item_id and source_turn | Stable id plus the conversation id and turn index the claim came from | M-014, conv-A turn 22 | Lets a grader check the agent cited the right turn, not a paraphrase from elsewhere |
| speaker_and_role | Who said it and in what capacity on the account | Customer contact, office manager | Authority depends on role; a visitor cannot change billing terms |
| claim_text and fact_type | The normalized statement and its type: account, preference, commitment, issue, third-party | Prefers email over phone; preference | Different types decay and conflict differently |
| record_time, valid_from, valid_to | When it was said, when it starts to hold, when it stops (or open) | Said day 0, valid day 0, open | Separates stale answers from correct historical answers |
| authority_basis | Why this speaker may state this fact for the account | Named account contact in the CRM | Grades whether the agent accepted a correction from the wrong person |
| permission_basis | The notice, contract term or request that allows keeping it, and for what purpose | Support notice covers service history | A fact can be true and still not be keepable for this purpose |
| retention_label and expiry_rule | The expected handling label (next section) and any end condition | RETAIN_UNTIL: after the callback date | This is the answer key |
| correction_history | Ordered list of later turns that changed the claim, with who changed it | Turn 3-7: switched to phone | Needed to score supersede behavior |
| deletion_scope | Which stores must drop the item on a deletion request: memory, summaries, vectors, logs | Memory store and session summaries | Deleting the chat alone may leave the memory behind |
| probe_and_expected_behavior | The later question or task, and what a correct agent does | Ask how to reach the contact; answer phone, cite turn 3-7 | Turns the label into a pass or fail check |
Which retention and forgetting labels does each item get?
Every candidate memory gets exactly one expected handling label, and the label is decided by a reviewer before any model sees the data. Seven labels cover the cases that matter in customer work; adding more usually means two reviewers will disagree on which one applies.
| Label | Meaning | Typical source in a first conversation | Pass condition at probe time |
|---|---|---|---|
| RETAIN | Keep and use until something changes it | Account identity, site address, product in use | Uses the fact and can name the turn it came from |
| RETAIN_UNTIL | Keep with an explicit end condition | Callback promised by a date; temporary contact while someone is away | Uses it before the end condition and stops after it |
| SUPERSEDE | Replaced by a later statement from someone with authority | Preference or contact changed in a later session | Uses the newer value and can say what it replaced |
| FORGET_ON_REQUEST | Must be dropped when the customer asks | Personal detail volunteered in passing | No longer used or restated after the request, in any store |
| DO_NOT_STORE | Should never have been written to memory | Speculation, a colleague's personal number, card details read aloud | Absent from memory from the first session onward |
| ABSTAIN | Never stated; the agent must say it does not know | A plausible fact the customer never mentioned | Declines or asks, instead of inventing an answer |
| CONFLICT_ESCALATE | Two sources disagree and neither clearly wins | Customer says one renewal month, the CRM record says another | Flags the conflict or asks, instead of silently picking one |
How do you write probes that catch stale or over-eager memory?
Write probes as later tasks, not quiz questions, and place them after the event that should change the answer. A probe asked before the correction only measures recall; the same probe asked after it measures whether memory moved with the world.
At a glance
How do you write probes that catch stale or over-eager memory
Public benchmarks give a useful vocabulary. LongMemEval tests five abilities, information extraction, multi-session reasoning, knowledge updates, temporal reasoning and abstention, over timestamped multi-session chat histories, and marks abstention questions separately because they have no answer location. LoCoMo annotates question answering over very long conversations with the dialog ids that contain each answer, which is the evidence pointer every probe here should carry.
Score the memory operations as well as the final answers. Mem0's default update prompt frames memory management as four operations, add, update, delete and no change, applied to each newly extracted fact. Logging which operation the agent chose for each labeled item shows where it failed: a stale answer caused by never updating looks the same at the surface as one caused by updating from the wrong speaker.
- Supersede probes: ask after the correction session, and require the newer value plus a pointer to the turn that changed it.
- Expiry probes: ask once before and once after the end condition; both must pass.
- Deletion probes: ask indirectly (draft a follow-up email) so the agent cannot pass by avoiding the literal topic.
- Abstention probes: use facts that are plausible for the account but never said; a confident answer is a failure.
- Authority probes: have a non-authorized speaker attempt a change; the correct behavior is to verify or decline.
Worked example (illustrative): a first support call and the three sessions after it
This example is illustrative, not drawn from a real customer. A facilities company calls a software vendor's support line for the first time. Over a 14-minute call the office manager explains that a scheduling feature is failing, says renewal will be decided by the operations director, asks for email rather than calls, mentions that a colleague handles invoices and reads out the colleague's mobile number, and says the team might move to a second site next spring.
Three later sessions test the memory. In session 2, eight days later, the office manager says calls are fine after all because email is going to spam. In session 3 the colleague asks the agent to change the renewal contact. In session 4 the office manager asks the vendor to stop storing the colleague's mobile number.
| Candidate memory | Label | Probe and timing | Correct behavior |
|---|---|---|---|
| Scheduling feature failing since last update | RETAIN_UNTIL (issue closed) | Session 2: what is still open? | Lists the issue as open, with its ticket |
| Prefers email | SUPERSEDE (session 2) | Session 3: draft outreach | Uses phone and notes the change came from the office manager |
| Operations director decides renewal | RETAIN; change attempt is an authority test | Session 3: colleague asks to become renewal contact | Verifies with the account contact before changing |
| Colleague's mobile number | DO_NOT_STORE (and FORGET_ON_REQUEST as backup) | Session 4: write an invoice reminder | Never had the number; if stored, it is gone from every store |
| Might move to a second site next spring | RETAIN_UNTIL, low confidence, valid_from in the future | Session 2: which site is the account at? | Answers the current site; mentions the possible move only as tentative |
| Number of seats on the plan | ABSTAIN | Session 3: how many seats do they have? | Says it was not discussed and checks the system of record |
Update, expire, delete or pause: which trigger fits each remembered fact?
Update when a speaker with authority corrects the fact; expire when its end condition passes; forget when the person asks or the purpose ends; stop using it, without deleting, when sources conflict until someone resolves them. Each of these is a separate behavior to grade, because agents that are good at one are often poor at another.
Deletion is the hardest to test because memory rarely lives in one place. OpenAI's help center, for example, says deleting a chat does not necessarily delete a saved memory created from it, and advises removing the information from the saved memory, the memory summary and the original chat. A deletion probe should therefore check every store the item could have reached: the memory store, session summaries, embeddings and any cached profile.
Where personal data of people in the EU is involved, the GDPR gives individuals a right to have inaccurate data rectified and, on listed grounds, a right to erasure without undue delay. That is general context, not legal advice; counsel decides which obligations apply to a given deployment. For the evaluation set, the practical consequence is that rectification and erasure need their own labeled items and their own probes, not a single forget test.
- Correction by an authorized speaker: SUPERSEDE, keep the old value in history, stop using it.
- End condition passed: expire the item; answer from the system of record if asked.
- Request to stop storing: delete across stores and confirm the scope covered.
- Speaker loses authority (left the company, contact changed): stop using their earlier account-level statements until reconfirmed.
- Conflict with the system of record: flag and ask; do not silently overwrite either side.
Where forgetting is the wrong answer, and what this set cannot show
The meaningful exception is the request not to be contacted, or not to be contacted in a certain way. The customer's other details may be deleted, but the suppression itself has to persist, otherwise the next session will happily reach out again. Label these as RETAIN with a note that they survive a general forget request, and write a probe that issues a deletion request first and a contact task second.
Over-forgetting is also a failure. An agent that drops the open issue because the conversation ended, or refuses to use an account fact because it might change, makes the customer repeat themselves. Weight false forgetting and false retention separately in the score so that a model cannot pass by remembering nothing.
Know the limits. A set built from a handful of first conversations tests behavior on those patterns, not on every customer type. Synthetic dialogues are easier than real ones: LoCoMo's repository includes the LLM-agent framework used to generate its long conversations, and LongMemEval compiles histories through an attribute-controlled pipeline, which gives clean ground truth but rarely captures messy authority, partial corrections or permissions. Report results per label, not only as one accuracy number, and state which stores the deletion probes covered.
Sourcing first conversations with real authority and correction history
The labels in this set depend on information synthetic chats do not have: who on the account had authority, what the support notice allowed, which later ticket corrected the first one. Real support and CRM histories carry that, which is why teams building memory evaluations often look for first contacts linked to the follow-up tickets, account records and deletion requests that came after them.
Memory test material of this kind can be requested from SourceX, which fills each request from U.S. businesses rather than from a standing inventory, working from the scope, volume, format, timing and licensing needs the buyer lists. The business that holds the conversations approves the scope and terms and signs before anything is handed over, and de-identification by SourceX through approved processing routes lets the linkage between sessions survive while personal details such as the colleague's mobile number in the example do not.
Why does agent memory turn dated change history into the scarce input?
Because the hard part of memory is change over time, and the evidence for grading it (what changed, when, on whose authority) sits in companies' own systems, not in the memory layer. That layer is turning into managed infrastructure that does not keep history for long. Amazon Bedrock AgentCore Memory keeps raw events for at most 365 days; each event's expiry is fixed when it is written, cannot be extended later, and expired events cannot be recovered. What can last longer are the long-term records a memory strategy extracts, which hold the agent's conclusion rather than the dated trail behind it.
Research agrees. MemoryAgentBench, accepted at ICLR 2026, scores conflict resolution, revising or removing stored information when new evidence contradicts it, as one of four memory competencies. A 2026 arXiv paper by Reddy and Challaram argues that these conflicts fail where answers are assembled after retrieval, not in storage, and that a deterministic freshness rule helps when facts carry explicit version markers; its revised version confines the demonstrated benefit to current-value conflicts with explicit version metadata. Benchmarks hand the agent that marker. A business has to recover it from timestamps scattered across systems, for permissions and promises as well as facts.
Which records date a changed fact, permission or promise, and how long do they last?
Field histories, consent logs, directory audit logs and order records date these changes, each on its own clock, and some clocks are short. Microsoft Entra ID keeps audit logs for seven days on its free edition and 30 days on P1 and P2 unless they are routed to storage, and expired logs cannot be recovered unless archived. Rule deadlines run on separate calendars: the FCC's 2024 robocall order sets a ceiling of 10 business days for honoring a revocation of consent, and the FTC's mail order rule turns a missed shipping date into a choice between the buyer's consent to the delay and a refund.
The agent-task input map below pairs each change with the state to hold, a checkable outcome, where dated evidence sits and what an export drops. Cells naming a rule or retention period are sourced; the rest are editorial assumptions to test:
| Change the agent must remember | State and constraint to hold | Outcome a grader can check | Where dated evidence sits, and its clock | What a records export alone drops |
|---|---|---|---|---|
| A fact corrected (address, preferred channel, billing contact) | Current value, the value it replaced and who changed it; only someone with authority may change it | The next action uses the new value and points to the change | CRM or ticket field history with editor and timestamp; retention differs by system and edition, so confirm it per source | Whether the agent learned of the change before or after it acted, which needs the session order |
| Consent to automated calls or texts withdrawn | No further automated contact with that person, and the opt-out survives any general forget request | No automated contact once the request is processed, and none past the deadline | Opt-out or consent log with the date the request arrived; FCC 24-24 allows no more than 10 business days to honor it, and the FCC took up revocation again in 2026 | Which messages went out after that date, usually held in a separate messaging system |
| Access to a source withdrawn | Facts read under the old access stay unused until access is rechecked | No answer reveals content the user can no longer open | Directory audit logs; Microsoft Entra ID keeps them 7 days (Free) or 30 days (P1, P2) unless archived | Who could see what on the session date, once the log window has closed |
| A dated promise (ship date, callback, refund) | The promise, who made it, its due date and whether a later delay was accepted | Promise kept, or a delay notice sent and the customer's answer recorded before the due date | Order and case records plus the conversation; under the FTC mail order rule a slipped ship date needs the buyer's consent or a refund, and silence counts only for a first delay of 30 days or less with a definite new date | Whether the promise was kept, what the delay cost and whether the customer agreed |
| A speaker's authority ended (contact left, new approver named) | Account-level statements from the former contact stop counting | Change requests from the former contact are verified, not applied | Contact role fields in the CRM, where their history is tracked, and directory changes | The date authority ended, which may be recorded nowhere |
Is the bottleneck the records, or the replay and outcomes around them?
Often the replay and the outcomes, which is why timestamped records alone make a weak asset. MemoryAgentBench splits reformulated datasets into chunks fed to the agent turn by turn and asks several questions of each long input. A records export supplies the chunks but not that harness, and it rarely shows whether acting on a stale memory cost anything.
Read as a value chain, temporal evidence has five links, and the business holds the three that are hard to copy:
- Dated change records from the system of record: held by the business and perishable, since logs age out on fixed windows.
- The permission trail behind each record (notice, consent, contract terms, withdrawals): held by the business and its counsel; without it nothing downstream is usable.
- As-of reconstruction of what the agent could have known on each session date: preparation work that depends on the first two links.
- A replay harness that feeds sessions in order and logs every memory write: software, and open-source versions such as MemoryAgentBench already exist.
- Outcome evidence (a promise kept, an opt-out honored, a customer made to repeat themselves): held by the business, often in a different system from the conversation.
- Decision: count a temporal dataset as an evaluation asset only when a sample shows the first, second and fifth links joined on the same cases; change records alone are a commodity input.
What would sink the temporal-evidence thesis, and what should investors ask first?
Three developments would weaken it. Engineering could close much of the gap: Reddy and Challaram treat freshness as a rule applied to versioned facts, not something learned from data, so spending may go to metadata pipelines rather than licensed histories. Tests rebuilt from earlier benchmarks, as MemoryAgentBench is, may satisfy evaluation teams. And supply may be thin: where audit logs last 30 days and memory events a year at most, a company can hold far less archived change history than its record count implies. The sources reviewed here publish no price or budget for temporal memory evaluation data, so demand is untested.
Ask any company offering such evidence, SourceX included: how far back each source system's change history reaches and whether it was archived before expiry; what share of sampled cases carry the permission trail and an outcome; whether opt-outs and promised dates link to the conversations behind them; how erasure requests were applied without losing the dated event; and what a memory evaluation built on the records showed against a benchmark-only baseline. SourceX licenses records from the businesses that hold them, each holder approving scope and terms and signing before delivery. It publishes no counts of such histories; investors who request materials through the investor page receive company figures directly.
Methods note: the input map and value chain were assembled on October 27, 2026 from the eight sources cited here. No dataset was sampled, no agent was run and no price was observed; retention figures are vendor defaults that archiving can extend, and the rule summaries are general information, not legal advice.
This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.
Sources for this section
- Amazon Bedrock AgentCore Memory retains raw short-term events for up to 365 days; an event's expiration is set at write time, cannot be extended later, and expired events cannot be recovered. Memory strategies process events into long-term memory records. Source
- MemoryAgentBench evaluates memory agents on four competencies, accurate retrieval, test-time learning, long-range understanding and conflict resolution, by reformulating earlier benchmarks and datasets into chunks that simulate multi-turn interaction, with two newly built sets (EventQA and FactConsolidation); it was accepted at ICLR 2026. Source
- The MemoryAgentBench paper describes conflict resolution as revising, overwriting or removing previously stored information when contradictory evidence arrives. Source
- Reddy and Challaram (arXiv, 2026) argue that memory conflict resolution fails in post-retrieval assembly rather than storage and propose deterministic aggregation when facts carry explicit version markers; their revised version limits the demonstrated benefit to current-value conflicts with explicit version metadata. Source
- Microsoft Entra ID retains audit logs for seven days on the Free edition and 30 days on P1 and P2, longer only if routed to storage, and expired data cannot be recovered unless it was archived. Source
- The FCC's 2024 order (FCC 24-24) requires revocations of consent to robocalls and robotexts, made by any reasonable means, to be honored within a reasonable time of no more than 10 business days after receipt. Source
- In 2026 the FCC took up consent revocation again (FCC 26-67), addressing whether callers may designate one exclusive means of revocation. Source
- The FTC's mail order rule requires a reasonable basis for a stated shipping time (30 days if none is stated); when a seller cannot ship on time it must obtain the buyer's consent to the delay or refund, and silence counts as consent only to a first delay notice with a definite revised date of 30 days or less. Source
Frequently asked questions
What is an agent memory evaluation dataset?
It is a set of labeled candidate memories taken from conversations, each with its source turn, speaker, times, permission and an expected handling label, plus later probes that check what the agent does with them. It grades retention, updating, expiry, deletion and abstention, not only recall.
Should an agent remember everything a customer says in the first conversation?
No. Speculation, one-off context, third-party details the speaker had no standing to share and sensitive details the task did not need should not be stored. Keeping them creates wrong answers later and widens what has to be deleted when someone asks.
How is this different from a long-context benchmark?
A long-context test puts the whole history in the prompt and asks a question. A memory evaluation checks what the agent chose to write, how it changed that record across sessions, and whether it stopped using a fact after a correction, an expiry or a deletion request.
Does deleting a conversation delete what the agent remembered from it?
Not necessarily. Some assistants store memories separately from chat history, so the source chat and the memory have to be removed separately. A good evaluation checks every store an item could reach, including summaries and embeddings.
How should abstention be scored?
Use facts that would be plausible for the account but were never stated, and count any confident answer as a failure. Accept answers that say the fact is unknown, ask the customer, or point to the system of record that would hold it.
How many items does a useful set need?
Enough per label to report each label separately with a stated uncertainty; a single overall score hides the failures that matter. Balance the labels deliberately, because real conversations produce far more RETAIN items than deletion or conflict cases.
Sources
- LongMemEval releases 500 questions testing five long-term memory abilities: information extraction, multi-session reasoning, knowledge updates, temporal reasoning and abstention. Source
- LoCoMo annotates question answering over very long conversations with the dialog ids that contain each answer, and its repository includes the LLM-agent framework used to generate conversations. Source
- Graphiti models each fact with a validity window and traces it to source episodes, invalidating contradicted facts while keeping their history. Source
- Graphiti's edge model stores valid_at, when a fact became true, and invalid_at, when it stopped being true. Source
- Mem0's default memory update prompt defines four operations for each new fact: add, update, delete and no change. Source
- OpenAI's help center says deleting a chat does not necessarily delete a saved memory created from it, and advises removing the information from each place it is kept. Source
- The GDPR sets out a right to rectification of inaccurate personal data (Article 16) and a right to erasure without undue delay on listed grounds (Article 17). Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.