AI uses for records
How to measure agent memory consolidation loss: a fidelity protocol
By SourceX Editorial · Updated
10 sections
Short answer
To measure what agent memory loses when it compresses long work histories, seed a month of business records with atomic facts of known source, valid time, authority and permission, run each memory method on the same history, and probe for omissions, distortions, stale values, lost history, false merges and permission or deletion failures against a sealed key. The page sets out the method only and reports no measured results.
Agent run
Agent memory consolidation
Key takeaways
- Loss can only be measured against an answer key written before compression runs, so seed the facts deliberately.
- Score omission, distortion, stale values, lost history, false merges, provenance, permission and deletion failures separately.
- Bound every comparison with a full-history reference and an evidence-only oracle that validates the probes.
- Measure memory contents for precision and recall, and answers for accuracy, abstention and leakage.
- Resample whole histories for confidence intervals and pre-register conditions, metrics and exclusions.
- This page is a method only; it reports no results.
How do you measure information lost when agent memory compresses long work histories?
Measure it against an answer key you wrote before compression ran. Seed a month of realistic work history with atomic facts whose source, valid time, authority and permission scope you know, let each memory method compress that history, then ask probes that can only be answered from those facts. Information loss is the difference between what the uncompressed history supports and what the compressed memory returns, broken down by kind of loss rather than collapsed into one score.
What follows is a measurement design rather than a finished study. It sets out the question, the population of histories, sampling, the conditions to compare, the metrics, how to express uncertainty and a pre-registration checklist. No results are reported here, and none of the counts below are findings; they are design parameters a team can change.
The protocol is written for AI lab teams and enterprise agent builders who need to decide whether a summarizing memory is safe for business work where a dropped exception or a stale value costs real time.
Why does compression lose information that matters?
Because every compression step decides what the agent will never see again. Anthropic describes compaction as summarizing a conversation that is nearing the context window limit and starting a new context window with the summary, and warns that overly aggressive compaction can drop subtle context whose importance only shows up later. Its API documentation for threshold compaction states that content blocks before the compaction block are dropped and the conversation continues from the summary.
Other designs compress differently but share the risk. MemGPT keeps a rolling queue of messages and stores, at the front of that queue, a recursive summary of messages that have been evicted; a summary of a summary is where a detail from week one is most likely to vanish. Mem0's default memory-update prompt tells the model to choose among adding, updating, deleting or leaving a memory unchanged, and to delete a memory when new facts contradict it, which keeps memory current but discards the earlier value an as-of question would need.
None of this says one design is worse. It says the losses differ in kind, so a single accuracy number would hide the trade-off a buyer actually cares about.
What counts as a loss, and how is each one detected?
Define loss types before collecting data, so the scoring cannot drift toward whatever the first run happens to show. Each type below gets its own probe set and its own metric.
| Loss type | What it looks like | Probe that detects it |
|---|---|---|
| Omission | A fact stated once in the month is absent from memory | Direct question whose only evidence is that fact |
| Distortion | The fact survives with a changed value, name or date | Same question; answer compared to the key exactly |
| Stale value | A superseded value is returned as current | Question about the current value after a correction |
| Lost history | The current value survives but earlier values are gone | As-of question about a date before the correction |
| False merge | Two similar entities or events are fused into one | Question that distinguishes the two |
| Provenance loss | The fact survives without its source record or author | Request for the record that supports the answer |
| Permission loss | A fact restricted to one role or customer becomes general | Same question asked from an unauthorized role |
| Resurrection | A fact deleted on request reappears after consolidation | Question about the deleted fact |
| Unsupported addition | The summary states something the history never said | Consistency check of memory content against the history |
Research question, population and sampling
Primary question: for a fixed month of business work, how much of each loss type does each compression method introduce, compared with an agent that can read the full uncompressed history? Secondary question: does loss concentrate in facts with particular properties, such as early position, single mention, later correction or restricted permission?
Population: work histories of the kind an operations agent would accumulate, for example a support queue, an account manager's CRM activity, a project coordinator's tickets and messages, or a finance close checklist. Each history covers one role over about 20 working days (a design choice) and includes the ordinary noise of real work: duplicates, status pings, forwarded threads and tool output.
Sampling: draw histories from at least two business types and several roles, stratified so that no single company or customer dominates. Within each history, seed facts deliberately rather than hoping natural ones appear: single-mention facts, facts corrected once or twice by an authorized person, facts restricted to one customer or role, facts withdrawn on request, and near-twin facts about similar entities. Spread each kind across early, middle and late weeks so position effects can be estimated. Keep the seeding rules and the answer key in a sealed file the memory systems never see.
Build the key in two passes. One person seeds facts and writes the expected answer, source record, valid-time range and permitted roles for each; a second person, working only from the history, answers every probe independently. Probes where the two disagree are rewritten or dropped before any memory system runs, so later errors can be blamed on memory rather than on an ambiguous key. Record the agreement rate between the two passes in the study report, because it sets a ceiling on how precisely any loss can be measured.
Which conditions should the fidelity experiment compare?
Every condition ingests the same histories in the same order and answers the same probes with the same model; only the memory route differs. Two reference conditions bound the result. The full-history condition reads everything uncompressed, where it fits, and shows what the model can do without compression. The evidence-only oracle reads just the records that contain the answer; if it fails a probe, the probe or the key is broken and that probe is dropped.
Public benchmarks already use this framing. LongMemEval provides an oracle variant containing only evidence sessions and labels question types including knowledge updates, temporal reasoning and abstention. LoCoMo evaluates retrieval-augmented answering using raw dialogs, extracted observations and session summaries as three alternative memory databases, which is exactly a comparison of compression levels.
| Condition | What the agent remembers | Role in the design |
|---|---|---|
| Full history | All records, uncompressed, in the prompt | Upper reference where the history fits |
| Evidence-only oracle | Only the records holding each answer | Probe validity check |
| Threshold compaction | A rolling summary created when context nears its limit | Tests summary-based compression |
| Recursive summary | Summaries of earlier summaries plus recent turns | Tests repeated compression |
| Extracted fact store | Atomic memories added, updated or deleted per turn | Tests extraction and overwrite |
| Session summaries plus retrieval | One summary per day or session, retrieved on demand | Tests coarse compression with search |
| Raw records plus retrieval | Uncompressed records, retrieved on demand | Separates retrieval failure from compression loss |
Which metrics separate recall from accuracy?
Report two families of measures, because a memory can be accurate and incomplete, or complete and wrong. FActScore, a method for long-form generation, makes the same split: it scores factual precision as the share of atomic facts in an output supported by a knowledge source, and treats the number of facts given as a separate indicator of recall. Apply both to memory contents directly, not just to answers.
Fidelity of the memory itself: decompose each compressed memory into atomic statements and count how many are supported by the history (precision) and how many seeded facts survive (recall). Check unsupported additions with a natural-language inference consistency model such as SummaC, which was built to detect summaries that are inconsistent with their source documents, and confirm a sample by hand.
Behavior at question time: accuracy per loss type and per position bucket; stale-answer rate on corrected facts; as-of accuracy; abstention rate on never-stated facts; provenance rate, meaning answers that cite the right source record; permission leakage count; and resurrection count after deletion. Report every rate with its denominator.
How should uncertainty and comparisons be reported?
Treat the history, not the probe, as the unit of sampling. Probes within one history share its quirks, so compute confidence intervals with a cluster bootstrap that resamples whole histories, and report the number of histories alongside the number of probes.
Use a paired design. Because every condition answers the same probes on the same histories, compare conditions with paired differences per probe, which removes much of the variation between histories. Run each condition several times with different seeds, since summarization and answering are both stochastic, and report run-to-run spread separately from sampling uncertainty.
Fix the comparisons in advance: each compression condition against raw records with retrieval, and each against the full-history reference. Correct for the number of comparisons, or report all of them with intervals and no significance claims. Report negative and null differences as fully as positive ones.
What must be fixed in writing before the first memory is compressed?
Write and timestamp these decisions before the first compressed memory is produced. Anything changed afterwards is reported as a deviation with its reason.
At a glance
What must be fixed in writing before the first memory is compressed
- Research questions and the single primary metric that will be headlined.
- History sources, roles, business types, time span and the stratification rule.
- Seeding rules: fact kinds, positions, number of corrections, permission scopes and deletions, plus who wrote the sealed answer key.
- Conditions, with model, memory settings, compaction thresholds and summary prompts frozen by version.
- Probe templates for each loss type and the rule for dropping probes the oracle fails.
- Scoring rules: exact-match fields, judge prompts and the human audit sample size for any model-based judgment.
- Unit of analysis, bootstrap method, number of runs per condition and planned comparisons.
- Exclusion rules for broken runs, and how infrastructure failures are counted.
- Data handling: where histories are stored, who may see them, de-identification steps and deletion after the study.
- What will be published: protocol version, code, probes and, where permitted, the histories.
When should an agent update, forget or stop using a remembered fact?
The experiment's probes double as an operating policy. Update when an authorized source supersedes the fact, but keep the prior value with its valid-time range so as-of questions still work. Forget when the person or system with authority withdraws it, and verify that the deletion survives the next consolidation. Stop using the fact, without deleting it, when its permission scope does not cover the current user or when its source record can no longer be found.
A memory design that cannot represent those three actions separately will show up in the results as stale answers, lost history or resurrections, which is the practical reason to score them separately.
Limits, and histories with known answers
Seeded facts are cleaner than real ones, so the protocol will understate losses that come from genuinely ambiguous records. Results from one model, one summary prompt and one month of work will not transfer to longer horizons without testing. Model-based scoring adds its own error, which is why the checklist requires a human audit sample. And because memory products change their prompts and defaults often, every result must be tied to the exact versions tested.
The hardest input to obtain is real work history with an answer key: records where corrections, withdrawals and permission limits are documented rather than inferred. Memory researchers can hand that search to SourceX, which works request by request and looks among U.S. businesses for matching records rather than keeping datasets in stock; scope, volume, language, format, timing and licensing needs go into the request-data form. No history is released until the business that holds it has signed an agreement on the scope.
Frequently asked questions
What is agent memory consolidation loss?
It is information an agent can no longer use after its memory compresses or rewrites earlier work, through summaries, compaction or fact extraction. It includes facts dropped entirely, facts altered, earlier values overwritten, and facts that lose their source or permission scope.
Does this page report how much information agents lose?
No. It publishes a measurement protocol only. Loss depends on the model, the memory design, the prompts and the histories, so results need a study run under a pre-registered design like this one.
Why compare against an evidence-only oracle?
The oracle reads only the records that contain each answer. If it fails a probe, the probe or the answer key is faulty, so that probe is removed instead of being counted as memory loss.
Is a bigger context window a substitute for testing memory?
No. A full-history condition shows what the model can do without compression, but histories outgrow windows, and compaction then decides what survives. The test is how much is lost when that happens.
What sample size does the experiment need?
Size it by number of histories, not probes, because probes inside one history are correlated. Use a pilot to estimate between-history variation, then choose enough histories for the confidence interval width you need on the primary metric.
Sources
- Anthropic's context engineering guide describes compaction as summarizing a conversation near the context window limit and reinitiating with the summary, and warns that overly aggressive compaction can lose subtle context. Source
- Anthropic's threshold compaction documentation states that the API drops all content blocks prior to the compaction block and continues the conversation from the summary. Source
- The MemGPT paper describes a FIFO message queue whose first entry holds a recursive summary of messages evicted from the queue. Source
- Mem0's default memory-update prompt has the model add, update, delete or leave memories unchanged, and delete a memory when retrieved facts contradict it. Source
- LongMemEval's README describes an oracle file with only evidence sessions and question types including knowledge-update, temporal-reasoning, multi-session and abstention. Source
- LoCoMo's README describes question-answering and event-summarization annotations and evaluates retrieval-augmented models using dialogs, observations and session summaries as databases. Source
- FActScore's README describes fine-grained atomic evaluation of factual precision and treats the number of atomic facts per response as an indicator of factual recall. Source
- SummaC (TACL) revisits NLI-based models for detecting inconsistency between summaries and their source documents. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.