Skip to content

AI uses for records

Business agent memory benchmark: a protocol for a month of history

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

This is a protocol, not a result. A business agent memory benchmark should replay a month of real, rights-cleared business events to the agent day by day, then ask questions whose answers depend on what it kept, updated or should have forgotten. Grade answers against evidence event IDs, treat leaking one customer's details into another's task as a hard failure, run repeated isolated trials, and pass every task through a privacy review before release.

Agent run

Business agent memory

This page covers business agent memory benchmark: a protocol for a month of history.Illustrative

Key takeaways

  • This page proposes a benchmark design and reports no results, scores or rankings.
  • Useful business memory is not recall alone: it is keeping commitments, replacing superseded facts, forgetting what must be forgotten and keeping customers apart.
  • Each task cites the evidence events that justify the answer, so grading checks why the agent knew something, not only what it said.
  • Leakage across customers or of deleted information is scored as a hard failure, separately from wrong answers.
  • Real month-long histories need a privacy review that keeps pseudonyms consistent across days, or the memory being tested disappears.

How could a benchmark test useful memory across a month of business history?

Feed the agent a month of business events in order, one day at a time, let it write whatever memory it uses, and then ask questions on later days whose correct answers depend on earlier events. Score whether it carried forward what mattered, replaced what changed, forgot what it was required to forget and kept each customer's information within that customer's tasks.

This page is the protocol for that business agent memory benchmark. It covers the research question, population and holdouts, ten task families, the task schema, the grader, a privacy review plan, baselines and a pre-registration checklist. It reports no measured findings, and every example is illustrative.

The research question is: after processing 30 days of events from one business function, does an agent answer later questions with the information that was current and permitted at the time of the question, cite the events that support it, and decline when it was never told?

What do existing memory benchmarks cover, and what does a business month add?

Existing long-term memory benchmarks test conversational recall well. LongMemEval releases 500 questions covering five abilities, information extraction, multi-session reasoning, knowledge updates, temporal reasoning and abstention, over timestamped chat histories, and offers an oracle version that contains only the evidence sessions. LoCoMo builds very long dialogues, averaging about 300 turns over up to 35 sessions, generated by language model agents grounded on personas and event graphs and then checked and edited by human annotators; its question answering annotations point to the dialogue turns that hold the evidence.

Business history adds four things those designs do not center on. The history is many records from many people, not one user talking to one assistant. Facts are replaced by later records, such as a revised delivery date or a new contact, and the old value must stop being used. Some information must be forgotten, because a record was deleted, a retention period ended or a customer asked. And memory must stay inside its scope: a support agent that mentions one customer's dispute while working another customer's ticket has failed even if every fact it stated was true.

Both benchmark designs are worth copying in two respects: evidence pointers on every question, and an oracle condition that separates retrieval failures from reasoning failures.

Population, sampling and holdout design

The unit is a 30-day window of events from one business function at one organization: a support queue, an account in a CRM, an accounts payable inbox or a field service dispatch board. Sample windows by function and by organization, cap each organization's share, and record why any window was excluded.

Replay is streaming. The harness delivers each day's events in time stamp order, lets the agent update its memory, and inserts questions at fixed points: mid-month, day 30 and a follow-up after a gap. Questions asked on day 30 about day 3 test retention; questions asked on day 30 about a fact changed on day 20 test updating.

Hold out whole organizations for a transfer test and whole task families for a generalization test, since an organization's vocabulary and habits repeat across its windows. Record the denominator: windows available, windows meeting inclusion rules, and windows dropped at rights review, de-identification or gold verification.

The longitudinal memory task suite

The suite is organized by what memory has to do, so results show which kind of memory fails. Examples are illustrative and describe question types, not real records.

The longitudinal memory task suite
Task familyMemory behavior testedIllustrative questionCorrect behavior
1. Carry-forward commitmentKeeping a promise made earlierDay 22: what did we tell this customer we would do by month end?State the commitment from day 4 and its due date
2. Superseded factReplacing an old valueDay 30: what is the delivery date for this order?Give the day 18 revision, not the original
3. Recurring patternNoticing repetition across daysHas this supplier sent duplicate invoices this month?Cite each duplicate pair
4. Temporal orderingPlacing events in sequenceDid the refund happen before or after the complaint was escalated?Answer with both event times
5. Cross-thread linkJoining related recordsIs this new ticket about the same outage as last week's?Link the two threads with evidence
6. Rule changeApplying the policy in forceWhich approval rule applied to the order on day 9?Use the version effective on day 9
7. Should-forgetNot using deleted or expired informationWhat phone number do we have for this contact?Not the number deleted on day 12
8. Scope isolationKeeping customers apartSummarize open issues for customer BNo details from customer A
9. AbstentionKnowing it was never toldWhat did the customer say about renewal?Say it is not in the history
10. Prospective reminderActing at the right later timeDay 25: anything due today?Raise the follow-up scheduled on day 11

The task schema

Every task is a pointer into a replayed window plus a question with a gold answer and the events that justify it. Keep gold fields out of the agent's input.

At a glance

The task schema

The task schema
FieldContentGrader use
task_id, family, splitStable ID, family number, public or holdoutPer-family reporting and split enforcement
window_id and ask_dayWhich 30-day window, and the replay day the question is askedFixes what the agent could have seen
scopeThe customer, account or case the question is aboutScope isolation check
questionNatural language request with required output formatInput
gold_answerStructured answer, or ABSTAINExact or rubric match
evidence_event_idsEvents that justify the gold answerEvidence check
superseded_event_idsEvents whose values must not be usedStale-answer check
forbidden_event_idsDeleted, expired or out-of-scope eventsLeakage hard failure
provenance and canarySource type, rights record, privacy review ID, canary stringRelease gate and contamination detection

The grader plan

Grade structure with code, free text with a calibrated rubric, and leakage with a hard rule. Anthropic's guidance on agent evaluations describes combining code-based, model-based and human graders, calibrating model graders against human experts, and starting each trial from a clean environment because shared state can inflate scores. Memory benchmarks are especially exposed to that last risk: a memory store left over from one trial answers the next one.

  • Answer check: structured fields such as dates, IDs and quantities are compared exactly; free-text answers go to a model grader calibrated on expert labels.
  • Evidence check: the answer must cite at least one evidence_event_id and no superseded_event_id; citing only the old value counts as a stale answer even when the wording looks right.
  • Leakage check: any content traceable to a forbidden_event_id, or to another scope, is a hard failure for the task, reported separately and never averaged away.
  • Abstention check: on family 9 tasks, ABSTAIN is correct and any answer is wrong; elsewhere, abstaining is scored as wrong and counted separately.
  • Repeated trials: run each window several times from a clean memory store and report pass^k, which tau-bench defines as the probability that all k independent trials succeed, next to single-trial accuracy, because an agent that remembers a commitment one time in three is not usable.

The privacy review plan

A month of real history is dense with personal and confidential information, and a memory benchmark cannot simply strip it, because consistent identities are what the agent has to remember. Run every window through the review below before it becomes a task. NIST SP 800-188 describes the governance pieces this plan borrows: choosing a data-sharing model, a disclosure review board, and re-identification testing to measure residual risk.

  • On release tiers: BIG-bench asks that any published task data carry its canary string so it can be excluded from training corpora; use the same practice, and keep the holdout private regardless.
The privacy review plan
StageWhat happensPass conditionOwner
1. Rights clearanceConfirm the license permits benchmark use, publication tiers and derived tasksWritten grant covers each intended release tierRights reviewer
2. MinimizationDrop fields no task needs, such as payment details and free-text notes unrelated to the functionEvery retained field maps to a task familyBenchmark lead
3. Consistent pseudonymsReplace names, emails and account numbers with stable pseudonyms across the whole windowSame person keeps one pseudonym on every dayData engineer
4. Date handlingShift dates by one offset per window, preserving weekdays and intervalsOrdering and gaps unchanged; gold answers recomputedData engineer
5. Free-text reviewDetect and replace identifiers in message bodies and attachmentsSample review finds no residual identifiersPrivacy reviewer
6. Should-forget setMark deleted, expired and opted-out records as forbidden rather than removing all traceForbidden events present only where a task tests forgettingBenchmark lead
7. Re-identification testAttempt linkage with public sources on a sampleResidual risk within the threshold set in the registrationPrivacy reviewer
8. Disclosure reviewBoard approves each window for a release tierSigned approval per tierDisclosure review board
9. Release controlsMark files with a canary string and keep a private holdoutCanary present; holdout never publishedBenchmark lead

Baselines, uncertainty and pre-registration

Use four baselines. A no-memory agent sees only the current day's events and shows how much each family needs memory at all. A full-context agent receives the whole window where it fits, and shows the ceiling when retrieval is not needed. An oracle condition gives only the evidence events, separating retrieval from reasoning. Human analysts with the same tools on a blind subset show achievable accuracy and expose ambiguous tasks.

At a glance

Baselines, uncertainty and pre-registration

Report accuracy, stale-answer rate, leakage rate and abstention precision and recall per family, each with a Wilson score interval as described in the NIST Engineering Statistics Handbook, and resample by organization rather than by task. Pre-register before any system is run, with the items below.

  • Research question, window length, ask days and output formats.
  • Population, functions, strata, caps and exclusion rules.
  • Holdout organizations and families.
  • Task schema, grader, rubric and calibration set versions.
  • Baselines, model identifiers, memory configurations and dates.
  • Number of trials per window and the pass^k values reported.
  • Metrics, interval method and resampling unit.
  • Privacy review thresholds and release tiers.
  • Conditions for publishing results, including disputed gold.

Which results would hold on companies the agent never saw?

Only results reported per family on held-out organizations and held-out families, with repeated-trial consistency and zero tolerance for leakage reported beside accuracy. A pooled score on random splits can reward an agent for learning one company's vocabulary rather than for remembering well. State the limits on the benchmark card: the functions sampled, the language of the records, a 30-day horizon, and the fact that synthetic tasks, where used, are labeled as such.

Teams building a benchmark like this need real month-long histories from operating businesses with rights that cover evaluation and publication tiers. SourceX does not keep datasets in stock; it sources them from U.S. businesses on request, and the request spells out scope, volume, language, format, timing and licensing needs.

Frequently asked questions

Does this page report how well any agent remembers?

No. It is a protocol for a benchmark that has not been run: task families, schema, grader, privacy review and a pre-registration checklist. Examples are illustrative, and results should follow only after grader calibration and with the holdout intact.

What should the longitudinal memory task and privacy review plan include?

Tasks that test carrying commitments forward, replacing superseded facts, forgetting deleted information, keeping customers apart and abstaining, each tied to evidence event IDs. The privacy plan covers rights clearance, minimization, consistent pseudonyms, date shifting, free-text review, re-identification testing, disclosure review and release controls.

Would a high memory score transfer to a company the agent never saw?

Per-family accuracy on organizations and families excluded from development, reported with repeated-trial consistency and leakage rates. Pooled accuracy on random splits mixes memory skill with familiarity with one organization's records.

Which baselines and intervals should a memory benchmark report?

Windows sampled across functions and organizations with caps, no-memory, full-context, oracle and human baselines, Wilson intervals per family, and resampling by organization. The registration should fix all of these before any system is run.

Why is leaking information a hard failure instead of a lower score?

In business use, one disclosure of another customer's details or of deleted information can outweigh many correct answers. Scoring it separately keeps a high accuracy number from hiding a privacy failure.

Sources

  • LongMemEval tests five long-term memory abilities with 500 questions over timestamped histories and offers an oracle evidence-only version. Source
  • LoCoMo dialogues average about 300 turns over up to 35 sessions, generated by agents and edited by human annotators. Source
  • LoCoMo's question answering annotations list the dialogue turns that contain the evidence. Source
  • tau-bench defines pass^k as the probability that all k independent trials succeed. Source
  • Agent evaluations combine code, model and human graders, calibrate model graders, and isolate each trial in a clean environment. Source
  • NIST SP 800-188 covers de-identification governance, including disclosure review boards and re-identification testing. Source
  • BIG-bench asks that published task data include its canary string. Source
  • The NIST handbook describes the Wilson score interval for a proportion. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies