Skip to content

Definitions and comparisons

Long context versus agent memory data: a matched evaluation

By SourceX Editorial · Updated

10 sections

Short answer

Long context capacity is how much text a model reads in one call; reliable agent memory is whether a fact learned earlier comes back later, across sessions, with its source, valid time and permissions intact, and is revised or withdrawn when the record changes. To tell them apart, run one history and one question set through a full-history prompt, a memory system and an evidence-only oracle, and score each failure type separately.

Agent run

Long context

This page covers long context versus agent memory data: a matched evaluation.Illustrative

Key takeaways

  • A window that accepts a long history can still miss facts in its middle, and it empties when the session ends.
  • Persistent memory fails differently: it serves superseded, unauthorized or withdrawn facts unless corrections reach the stored copy.
  • A fair test feeds the same timestamped history to a long-context arm, a memory arm and an evidence-only oracle.
  • Every remembered fact needs a source record, authority, valid time, permission scope, supersession link and deletion state.
  • Report accuracy by probe type and position, plus stale-answer, deletion-leakage and scope-leakage counts, never one blended score.

How do you distinguish long context capacity from reliable agent memory?

Long context capacity is how much text a model can accept in one call. Reliable agent memory is whether a fact learned at one point is retrieved later, in a different session, with its source, its valid time and its permissions intact, and whether it is revised or withdrawn when the record changes. You tell them apart by asking the same questions about the same facts through two delivery routes and scoring where each one fails.

The two properties fail in different ways. A long window can hold an entire account history and still miss a fact buried in the middle of it. A memory store can hold one tidy note about that fact and still serve it after it was corrected, because nothing told the store the old note was superseded. A buyer who only measures one of them will ship an agent with the other weakness.

This guide gives a matched evaluation design: one history, one question set, three arms (full history in the prompt, a persistent memory system, and an evidence-only oracle), and probes chosen to separate recall from currency, authority and deletion.

Why a larger context window does not answer the memory question

Advertised window size is a ceiling, not a measured ability. The RULER benchmark from NVIDIA reports that while every model it tested claims a context of 32K tokens or more, only about half held their quality at 32K, and almost all fell below its threshold before reaching their claimed length. Position matters too: the Lost in the Middle study found that models often do best when the relevant passage sits at the beginning or end of the input and degrade when it sits in the middle.

Even a perfect window ends when the session ends. Anything an agent should know next week has to be written somewhere outside the prompt and read back. Anthropic's documentation for its memory tool describes exactly that split: the model asks for file operations in a memory directory that the developer's own application stores, so information can persist between sessions without occupying the context window. Once facts live outside the window, their correctness depends on what was written, when, by whom, and whether later changes reached the stored copy.

So the question for a buyer of evaluation or training data is not whether the window is big enough. It is whether the agent answers correctly when the fact it needs is old, has changed, came from a less authoritative source, or should no longer be used at all.

What must a remembered business fact carry to be checkable?

Each fact in the test history needs enough metadata that a grader can decide, without judgment calls, what the right answer is at any probe time. Plain text notes are not enough, because the grader then has to infer validity from prose.

The fields below follow two published patterns. Graphiti, an open-source temporal graph library for agent memory, stores separate valid_at and invalid_at times on each fact edge, plus an expired_at time for when the record itself was invalidated. The W3C PROV ontology gives standard terms for derivation and attribution (wasDerivedFrom, wasAttributedTo), which map cleanly to source record and asserting party.

What must a remembered business fact carry to be checkable?
FieldWhat it holdsWhy the grader needs it
fact_id and statementA stable id and one atomic claim, for example the account's billing contactLets every probe point to exactly one expected fact
source_recordSystem and record id the fact came from, such as a CRM contact or a ticket commentSeparates a fact the agent was told from one it inferred
asserted_by and authorityWho stated it and whether that role may change this fieldDecides which of two conflicting claims wins
valid_from and valid_toWhen the fact became true and stopped being true in the businessAnswers as-of questions without guessing
recorded_atWhen the fact entered the history the agent seesSeparates event time from learning time
permission_scopeWhich users, tenants or purposes may receive the factTurns leakage into a scorable failure
supersedes and correction_noteThe fact_id this one replaces and whyMakes stale answers detectable
deletion_stateActive, withdrawn on request, or expired by retention ruleTests whether forgotten facts stay forgotten

The matched long context and persistent memory evaluation

The core of the design is matching. Every arm sees the same sessions in the same order and answers the same probes; only the route by which the history reaches the model differs. LongMemEval, a public benchmark for chat assistant memory, already uses this shape: its S variant is built so that the whole history (roughly 40 sessions, about 115k Llama 3 tokens) can be concatenated into a long-context baseline, its oracle variant keeps only the evidence sessions, and its M variant, at roughly 500 sessions, is too long for long-context testing at all.

Copy that structure with business histories. Arm A puts the full history into the prompt. Arm B lets the agent's memory system ingest sessions one at a time and answers from what it stored. Arm C, the oracle, gets only the sessions that contain the evidence. If Arm C fails, the probe or the answer key is broken; if C passes and A fails, the long window is the weak point; if A passes and B fails, the memory pipeline lost or corrupted the fact.

The matched long context and persistent memory evaluation
Probe typeWhat it isolatesSetup in the historyPass condition
Static recallBasic retrieval across sessionsFact stated once, never changedCorrect value with its source record
Updated factCurrency after a correctionFact stated, later superseded by an authorized partyNewest valid value; old value not offered
As-of questionValid time reasoningSame as updated fact, probe asks about an earlier dateValue valid on that date, not today's
Conflicting authoritySource rankingTwo parties assert different values, only one may change the fieldValue from the authorized party, conflict noted
Withdrawn factDeletion behaviorFact removed on request in a later sessionAgent declines to use it
Out-of-scope factPermission boundaryFact visible to one tenant or role, probe asked as anotherNo disclosure
Never statedAbstentionPlausible fact that does not appear anywhereAgent says it does not know
Position controlWindow placement effectsSame static fact placed early, middle or lateEqual accuracy across positions

How to build a history that is fair to both arms

Fairness comes from controlling the variables you are not testing. Hold the total token count of each history below the effective length you measured for the model under test, not its advertised length, so Arm A is not set up to fail on size alone. Then vary only position and age.

Place each evidence turn in a position bucket (early, middle, late) and an age bucket (sessions since the fact was recorded), and balance probe types across the buckets. LoCoMo, the ACL 2024 benchmark for very long-term conversational memory, shows the bookkeeping that makes this possible: every session carries a date and time, every turn has a dialog id, and each question lists the dialog ids that contain its answer as evidence.

Fill the rest of each session with realistic distractors from the same business domain: other accounts, similar names, near-duplicate tickets. Generic chit-chat filler makes retrieval too easy. Freeze the history, the probe list and the answer key before running any arm, and record the probe time for every question, because the correct answer to an updated-fact probe depends on when it is asked.

Checklist · 4 items

  1. Write each history as timestamped sessions with turn ids, and keep the fact register from the previous section alongside it.
  2. Generate the answer key from the fact register, not from reading the transcript.
  3. Run the oracle arm first and drop or fix any probe it fails.
  4. Version the history, the register and the key together so a rerun uses the same three files.

0 of 4 ticked. Tick each item as you go.

Which measures show where memory breaks?

Report accuracy per probe type, never one blended score. A memory system can look strong on static recall while serving stale values on every updated fact, and the average hides it.

Alongside accuracy, track the failure modes that matter in a business setting. The stale-answer rate counts probes where the agent offered a superseded value. Deletion leakage counts withdrawn facts that resurfaced. Scope leakage counts out-of-scope disclosures. Citation correctness checks whether the agent named the right source record when it answered. For the long-context arm, break accuracy down by position bucket to see whether the middle of the history is the weak spot.

  • Accuracy by probe type and arm, with the oracle as the ceiling.
  • Stale-answer rate on updated-fact and as-of probes.
  • Deletion leakage and scope leakage counts, reported as raw counts as well as rates.
  • Abstention precision: how often a refusal was correct.
  • Position and age breakdowns for every arm.
  • Uncertainty from repeated runs and item-level resampling, reported as intervals rather than single numbers.
  • Token and latency cost per probe as a secondary column, never as the headline.

Worked example: a billing contact that changed twice

This case is illustrative and invented to show how the answer key works. A software vendor's support history covers one customer account across 16 sessions. In session 2 the customer's finance lead names a billing contact. In session 9 the customer's account administrator replaces that contact. In session 12 a sales rep on the vendor side mentions the original contact again in passing. In session 15 the customer asks the vendor to stop using the second contact's mobile number.

The register records four facts: the original contact (valid from session 2 to session 9), the replacement (valid from session 9), the sales rep's mention (recorded, but the rep has no authority over the customer's billing contact), and the withdrawal of the mobile number (from session 15).

  • Exception: if every probe in your real workload is answered within one session and the history fits comfortably inside the measured window, Arm A may be the better design, because summarizing into memory can drop details a full transcript keeps.
  • Limit: the example has one account and four facts. A real test set needs many accounts, each with its own register, so results do not hinge on one storyline.
Worked example: a billing contact that changed twice
Probe asked afterQuestionExpected answerTypical failure to watch for
Session 10Who is the billing contact?The replacement named in session 9Memory arm returns the session 2 contact because the update was stored as a new note, not a supersession
Session 13Who is the billing contact?Still the replacement; the session 12 mention carries no authorityEither arm treats the most recent mention as the newest truth
Session 13Who was the billing contact in the first week?The original contact, valid until session 9Agent answers with today's value
Session 16What number should we call the billing contact on?Not the withdrawn mobile; use another channel on record or askMemory arm still holds the number; long context arm reads it from session 9

When should a stored fact be revised, expired or withheld?

Treat every remembered fact as a dated claim from a named source, not as settled knowledge. That framing gives four clear triggers. Revise it when a party with authority over that field states a new value. Expire it when its valid_to passes or a retention rule ends it. Withhold it when the requester is outside its permission scope. Withdraw it when the person or business it concerns asks for it to stop being used.

Withdrawal is the trigger most often mis-implemented, because deleting the conversation and deleting the memory are separate actions in many products. OpenAI's help center, for example, says that deleting a chat does not remove a saved memory that came from it; both have to be removed. A test set should therefore include withdrawals issued against the memory as well as against the source transcript, and check that neither route lets the fact return.

Conflicts deserve their own rule. When two sources disagree and neither has clear authority, the right behavior is usually to surface the conflict and ask, not to pick the newer one. Score that as a pass in the answer key, or the evaluation will reward confident guessing.

Where the matched comparison stops being informative

The design measures retrieval and currency of discrete facts. It says little about procedural memory (how the agent learned to do a task better) or about summaries whose correctness is a matter of judgment. Keep those out of the probe set or score them separately with human review.

It also depends on the answer key being right. Facts drawn from real systems often have unclear authority (who is allowed to change a shipping address in a shared account?) and fuzzy valid times. When the business records owner cannot state the rule for a field, drop that field from the test rather than guessing. Bring in an agent memory engineer and the owner of the source system to sign off on the register before scoring. If the facts involve personal data, a privacy reviewer should confirm that withdrawn and out-of-scope facts were handled in the test data itself, not just in the agent.

Sourcing account histories with real corrections

Synthetic histories make the bookkeeping easy but rarely contain the messy part: corrections from the wrong person, partial withdrawals, and facts restated with small differences months apart. Real support, CRM and account-management records contain all of these, along with the system timestamps that let you set valid times without guessing.

Histories like these are found for a buyer rather than pulled from inventory, because SourceX keeps none: the buyer lists scope, volume, language, format, timing and licensing terms, and SourceX approaches U.S. businesses whose account records fit. It carries out qualification, rights review, contracting, preparation and delivery, and the supplier must approve the scope and sign before a single record moves. A request for memory-evaluation histories should name the fact types, the correction and withdrawal events, and the timestamp fields you need in the register.

Frequently asked questions

Is a bigger context window a substitute for agent memory?

No. A window only holds what is in the current call, and quality often drops well before the advertised length. Anything needed in a later session has to be stored and read back, and that storage can go stale or leak in ways a window test never shows.

What is the difference between event time and recorded time for a remembered fact?

Event time is when the fact became true in the business, such as the date a contact changed. Recorded time is when the agent first saw it. Keeping both lets you grade as-of questions and spot an agent that treats late news as new news.

How many probes should a matched memory evaluation have?

There is no fixed number. Make sure every probe type appears in every position and age bucket for every arm, and run enough repetitions that the interval around each per-type score is narrow enough to support the decision you are making.

Why include an oracle arm?

The oracle sees only the evidence sessions, so it shows whether a probe is answerable at all. If the oracle fails, fix the probe or the answer key before blaming the long-context or memory arm.

Should a memory system ever prefer an older fact over a newer one?

Yes, when the newer statement came from someone without authority over that field, or when the question asks about an earlier date. Recency is a tiebreaker only between equally authoritative sources.

Sources

  • RULER reports that almost all tested models fall below its quality threshold before reaching their claimed context lengths. Source
  • Lost in the Middle (TACL 2024) finds performance is often highest when relevant information sits at the start or end of the input and degrades in the middle. Source
  • LongMemEval provides a long-context baseline, an oracle variant with only evidence sessions, and an M variant too long for long-context testing. Source
  • LoCoMo timestamps every session, gives every turn a dialog id and lists evidence dialog ids for each question. Source
  • Anthropic's memory tool is client-side: the developer's application stores memory files so information persists between sessions outside the context window. Source
  • OpenAI's Memory FAQ says deleting a chat does not remove a saved memory that came from it. Source
  • Graphiti stores valid_at, invalid_at and expired_at timestamps on each fact edge. Source
  • The W3C PROV ontology defines wasDerivedFrom (entity to entity) and wasAttributedTo (entity to responsible agent). Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies