AI uses for records
Business agent memory benchmark: a protocol for a month of history
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
This is a protocol, not a result. A business agent memory benchmark should replay a month of real, rights-cleared business events to the agent day by day, then ask questions whose answers depend on what it kept, updated or should have forgotten. Grade answers against evidence event IDs, treat leaking one customer's details into another's task as a hard failure, run repeated isolated trials, and pass every task through a privacy review before release.
Agent run
Business agent memory
Key takeaways
- This page proposes a benchmark design and reports no results, scores or rankings.
- Useful business memory is not recall alone: it is keeping commitments, replacing superseded facts, forgetting what must be forgotten and keeping customers apart.
- Each task cites the evidence events that justify the answer, so grading checks why the agent knew something, not only what it said.
- Leakage across customers or of deleted information is scored as a hard failure, separately from wrong answers.
- Real month-long histories need a privacy review that keeps pseudonyms consistent across days, or the memory being tested disappears.
How could a benchmark test useful memory across a month of business history?
Feed the agent a month of business events in order, one day at a time, let it write whatever memory it uses, and then ask questions on later days whose correct answers depend on earlier events. Score whether it carried forward what mattered, replaced what changed, forgot what it was required to forget and kept each customer's information within that customer's tasks.
This page is the protocol for that business agent memory benchmark. It covers the research question, population and holdouts, ten task families, the task schema, the grader, a privacy review plan, baselines and a pre-registration checklist. It reports no measured findings, and every example is illustrative.
The research question is: after processing 30 days of events from one business function, does an agent answer later questions with the information that was current and permitted at the time of the question, cite the events that support it, and decline when it was never told?
What do existing memory benchmarks cover, and what does a business month add?
Existing long-term memory benchmarks test conversational recall well. LongMemEval releases 500 questions covering five abilities, information extraction, multi-session reasoning, knowledge updates, temporal reasoning and abstention, over timestamped chat histories, and offers an oracle version that contains only the evidence sessions. LoCoMo builds very long dialogues, averaging about 300 turns over up to 35 sessions, generated by language model agents grounded on personas and event graphs and then checked and edited by human annotators; its question answering annotations point to the dialogue turns that hold the evidence.
Business history adds four things those designs do not center on. The history is many records from many people, not one user talking to one assistant. Facts are replaced by later records, such as a revised delivery date or a new contact, and the old value must stop being used. Some information must be forgotten, because a record was deleted, a retention period ended or a customer asked. And memory must stay inside its scope: a support agent that mentions one customer's dispute while working another customer's ticket has failed even if every fact it stated was true.
Both benchmark designs are worth copying in two respects: evidence pointers on every question, and an oracle condition that separates retrieval failures from reasoning failures.
Population, sampling and holdout design
The unit is a 30-day window of events from one business function at one organization: a support queue, an account in a CRM, an accounts payable inbox or a field service dispatch board. Sample windows by function and by organization, cap each organization's share, and record why any window was excluded.
Replay is streaming. The harness delivers each day's events in time stamp order, lets the agent update its memory, and inserts questions at fixed points: mid-month, day 30 and a follow-up after a gap. Questions asked on day 30 about day 3 test retention; questions asked on day 30 about a fact changed on day 20 test updating.
Hold out whole organizations for a transfer test and whole task families for a generalization test, since an organization's vocabulary and habits repeat across its windows. Record the denominator: windows available, windows meeting inclusion rules, and windows dropped at rights review, de-identification or gold verification.
The longitudinal memory task suite
The suite is organized by what memory has to do, so results show which kind of memory fails. Examples are illustrative and describe question types, not real records.
| Task family | Memory behavior tested | Illustrative question | Correct behavior |
|---|---|---|---|
| 1. Carry-forward commitment | Keeping a promise made earlier | Day 22: what did we tell this customer we would do by month end? | State the commitment from day 4 and its due date |
| 2. Superseded fact | Replacing an old value | Day 30: what is the delivery date for this order? | Give the day 18 revision, not the original |
| 3. Recurring pattern | Noticing repetition across days | Has this supplier sent duplicate invoices this month? | Cite each duplicate pair |
| 4. Temporal ordering | Placing events in sequence | Did the refund happen before or after the complaint was escalated? | Answer with both event times |
| 5. Cross-thread link | Joining related records | Is this new ticket about the same outage as last week's? | Link the two threads with evidence |
| 6. Rule change | Applying the policy in force | Which approval rule applied to the order on day 9? | Use the version effective on day 9 |
| 7. Should-forget | Not using deleted or expired information | What phone number do we have for this contact? | Not the number deleted on day 12 |
| 8. Scope isolation | Keeping customers apart | Summarize open issues for customer B | No details from customer A |
| 9. Abstention | Knowing it was never told | What did the customer say about renewal? | Say it is not in the history |
| 10. Prospective reminder | Acting at the right later time | Day 25: anything due today? | Raise the follow-up scheduled on day 11 |
The task schema
Every task is a pointer into a replayed window plus a question with a gold answer and the events that justify it. Keep gold fields out of the agent's input.
At a glance
The task schema
| Field | Content | Grader use |
|---|---|---|
| task_id, family, split | Stable ID, family number, public or holdout | Per-family reporting and split enforcement |
| window_id and ask_day | Which 30-day window, and the replay day the question is asked | Fixes what the agent could have seen |
| scope | The customer, account or case the question is about | Scope isolation check |
| question | Natural language request with required output format | Input |
| gold_answer | Structured answer, or ABSTAIN | Exact or rubric match |
| evidence_event_ids | Events that justify the gold answer | Evidence check |
| superseded_event_ids | Events whose values must not be used | Stale-answer check |
| forbidden_event_ids | Deleted, expired or out-of-scope events | Leakage hard failure |
| provenance and canary | Source type, rights record, privacy review ID, canary string | Release gate and contamination detection |
The grader plan
Grade structure with code, free text with a calibrated rubric, and leakage with a hard rule. Anthropic's guidance on agent evaluations describes combining code-based, model-based and human graders, calibrating model graders against human experts, and starting each trial from a clean environment because shared state can inflate scores. Memory benchmarks are especially exposed to that last risk: a memory store left over from one trial answers the next one.
- Answer check: structured fields such as dates, IDs and quantities are compared exactly; free-text answers go to a model grader calibrated on expert labels.
- Evidence check: the answer must cite at least one evidence_event_id and no superseded_event_id; citing only the old value counts as a stale answer even when the wording looks right.
- Leakage check: any content traceable to a forbidden_event_id, or to another scope, is a hard failure for the task, reported separately and never averaged away.
- Abstention check: on family 9 tasks, ABSTAIN is correct and any answer is wrong; elsewhere, abstaining is scored as wrong and counted separately.
- Repeated trials: run each window several times from a clean memory store and report pass^k, which tau-bench defines as the probability that all k independent trials succeed, next to single-trial accuracy, because an agent that remembers a commitment one time in three is not usable.
The privacy review plan
A month of real history is dense with personal and confidential information, and a memory benchmark cannot simply strip it, because consistent identities are what the agent has to remember. Run every window through the review below before it becomes a task. NIST SP 800-188 describes the governance pieces this plan borrows: choosing a data-sharing model, a disclosure review board, and re-identification testing to measure residual risk.
- On release tiers: BIG-bench asks that any published task data carry its canary string so it can be excluded from training corpora; use the same practice, and keep the holdout private regardless.
| Stage | What happens | Pass condition | Owner |
|---|---|---|---|
| 1. Rights clearance | Confirm the license permits benchmark use, publication tiers and derived tasks | Written grant covers each intended release tier | Rights reviewer |
| 2. Minimization | Drop fields no task needs, such as payment details and free-text notes unrelated to the function | Every retained field maps to a task family | Benchmark lead |
| 3. Consistent pseudonyms | Replace names, emails and account numbers with stable pseudonyms across the whole window | Same person keeps one pseudonym on every day | Data engineer |
| 4. Date handling | Shift dates by one offset per window, preserving weekdays and intervals | Ordering and gaps unchanged; gold answers recomputed | Data engineer |
| 5. Free-text review | Detect and replace identifiers in message bodies and attachments | Sample review finds no residual identifiers | Privacy reviewer |
| 6. Should-forget set | Mark deleted, expired and opted-out records as forbidden rather than removing all trace | Forbidden events present only where a task tests forgetting | Benchmark lead |
| 7. Re-identification test | Attempt linkage with public sources on a sample | Residual risk within the threshold set in the registration | Privacy reviewer |
| 8. Disclosure review | Board approves each window for a release tier | Signed approval per tier | Disclosure review board |
| 9. Release controls | Mark files with a canary string and keep a private holdout | Canary present; holdout never published | Benchmark lead |
Baselines, uncertainty and pre-registration
Use four baselines. A no-memory agent sees only the current day's events and shows how much each family needs memory at all. A full-context agent receives the whole window where it fits, and shows the ceiling when retrieval is not needed. An oracle condition gives only the evidence events, separating retrieval from reasoning. Human analysts with the same tools on a blind subset show achievable accuracy and expose ambiguous tasks.
At a glance
Baselines, uncertainty and pre-registration
Report accuracy, stale-answer rate, leakage rate and abstention precision and recall per family, each with a Wilson score interval as described in the NIST Engineering Statistics Handbook, and resample by organization rather than by task. Pre-register before any system is run, with the items below.
- Research question, window length, ask days and output formats.
- Population, functions, strata, caps and exclusion rules.
- Holdout organizations and families.
- Task schema, grader, rubric and calibration set versions.
- Baselines, model identifiers, memory configurations and dates.
- Number of trials per window and the pass^k values reported.
- Metrics, interval method and resampling unit.
- Privacy review thresholds and release tiers.
- Conditions for publishing results, including disputed gold.
Which results would hold on companies the agent never saw?
Only results reported per family on held-out organizations and held-out families, with repeated-trial consistency and zero tolerance for leakage reported beside accuracy. A pooled score on random splits can reward an agent for learning one company's vocabulary rather than for remembering well. State the limits on the benchmark card: the functions sampled, the language of the records, a 30-day horizon, and the fact that synthetic tasks, where used, are labeled as such.
Teams building a benchmark like this need real month-long histories from operating businesses with rights that cover evaluation and publication tiers. SourceX does not keep datasets in stock; it sources them from U.S. businesses on request, and the request spells out scope, volume, language, format, timing and licensing needs.
Frequently asked questions
Does this page report how well any agent remembers?
No. It is a protocol for a benchmark that has not been run: task families, schema, grader, privacy review and a pre-registration checklist. Examples are illustrative, and results should follow only after grader calibration and with the holdout intact.
What should the longitudinal memory task and privacy review plan include?
Tasks that test carrying commitments forward, replacing superseded facts, forgetting deleted information, keeping customers apart and abstaining, each tied to evidence event IDs. The privacy plan covers rights clearance, minimization, consistent pseudonyms, date shifting, free-text review, re-identification testing, disclosure review and release controls.
Would a high memory score transfer to a company the agent never saw?
Per-family accuracy on organizations and families excluded from development, reported with repeated-trial consistency and leakage rates. Pooled accuracy on random splits mixes memory skill with familiarity with one organization's records.
Which baselines and intervals should a memory benchmark report?
Windows sampled across functions and organizations with caps, no-memory, full-context, oracle and human baselines, Wilson intervals per family, and resampling by organization. The registration should fix all of these before any system is run.
Why is leaking information a hard failure instead of a lower score?
In business use, one disclosure of another customer's details or of deleted information can outweigh many correct answers. Scoring it separately keeps a high accuracy number from hiding a privacy failure.
Sources
- LongMemEval tests five long-term memory abilities with 500 questions over timestamped histories and offers an oracle evidence-only version. Source
- LoCoMo dialogues average about 300 turns over up to 35 sessions, generated by agents and edited by human annotators. Source
- LoCoMo's question answering annotations list the dialogue turns that contain the evidence. Source
- tau-bench defines pass^k as the probability that all k independent trials succeed. Source
- Agent evaluations combine code, model and human graders, calibrate model graders, and isolate each trial in a clean environment. Source
- NIST SP 800-188 covers de-identification governance, including disclosure review boards and re-identification testing. Source
- BIG-bench asks that published task data include its canary string. Source
- The NIST handbook describes the Wilson score interval for a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.