AI uses for records
Is your agent benchmark testing memory? An enterprise benchmark leakage audit
By SourceX Editorial · Updated
10 sections
Short answer
An agent benchmark tests memory instead of ability when its test tasks share files, companies, entities, templates, time periods or environment state with what the model or its prompts already saw. To tell, run a leakage audit on each of those axes, rebuild the test set from groups absent from training and dated after the training cutoff, and score each held-out view separately. A large drop shows the original score rewarded recall.
Evaluation split
Enterprise benchmark leakage
Key takeaways
- Business records come in groups (companies, customers, vendors, templates), so a random split lets a model score well by remembering the group.
- Choose the split unit from the claim: company for new customers, entity for new counterparties, time for future use.
- Exact-match deduplication misses paraphrased and translated items; pair it with embedding search and reviewed matches.
- The most common hidden leak is an answer sitting in a later reply, attachment, back-filled field or the task environment itself.
- Overlap is acceptable for in-distribution benchmarks only when labeled as such and reported separately from transfer claims.
What does leakage look like in an enterprise agent benchmark?
Leakage means some information that would not be available in real use reaches the model, its prompts or its developers before the test, so the score measures recall of that information rather than the skill the benchmark claims to test. In enterprise agent benchmarks the leak is rarely a copied test file. It is usually the same company, the same customer, the same document template or the same month showing up on both sides of the split.
Kapoor and Narayanan surveyed machine learning work across many scientific fields, found leakage in 17 of them and organized the failures into a taxonomy of eight leakage types, from textbook errors such as no held-out test set to subtle ones such as dependence between training and test samples and test data drawn from a different time than the claim. The same failure modes apply when the test items are support cases, invoices or tickets instead of scientific measurements.
Agent benchmarks add two routes that classic supervised tests do not have. The environment itself can carry the answer, for example a database snapshot that already contains the resolved record, and the task wording can be generated from a small set of templates, so a model that has seen ten variants of a template can solve the eleventh by pattern alone. A leakage audit has to look at all of these routes, not only at exact text overlap.
Why is a random split the wrong default for business records?
A random split assumes every record is independent, and business records almost never are. Five invoices from one vendor, twenty tickets from one customer or a hundred tasks generated from one workflow template are a group: they share names, formats, policies and often answers. Put some in training and some in test, and the model can score well by remembering the group.
The scikit-learn documentation states the general rule plainly: the independence assumption breaks when the process that generates the data yields groups of dependent samples, and to measure generalization to unseen groups every sample in the validation fold must come from groups not represented at all in the paired training fold. For records that change over time, the same documentation warns that ordinary shuffled cross-validation creates unreasonable correlation between training and test instances, and that models should be evaluated on future observations.
So the first decision in any enterprise benchmark is the unit of the split. For a model that will be sold to new customers, split by company or tenant. For a model that will meet new counterparties inside one company, split by entity such as customer, vendor or employee. For a model that will run next quarter, split by time, with every test item dated after every training item. When the deployment faces all three, the clean test set is the intersection: new companies, new entities and later dates.
What should the leakage audit across files, entities and task variants include?
The audit below is the asset this page exists to provide. Run it once when the benchmark is built and again every time items, environments or training data change. Each row names the leakage axis, what leaks along it, the check to run, a pass rule and the evidence to file with the benchmark card so a reviewer can repeat the result.
Write the pass rules down before running any check. Thresholds chosen after seeing the overlap tend to drift toward whatever makes the benchmark look clean. Name one owner per axis, because entity resolution, template clustering and access review usually sit with different people, and an axis without an owner is the one that gets skipped.
Order matters. Fix lineage, company and time first, since those are cheap exact joins that remove whole groups. Then run entity and template clustering on what remains, and finish with near-duplicate search and environment diffs, which are the most expensive checks and the ones most likely to need human judgment.
| Axis | What leaks | Check to run | Pass rule and evidence to file |
|---|---|---|---|
| Source lineage (files) | The same source document, export or attachment feeds both a training item and a test item | Hash every source file and record which items it produced; join test lineage to training lineage | Zero shared file hashes; file the lineage table with file hash, item ids and split |
| Company or tenant | Formats, policies and vocabulary specific to one organization | Group items by company id and confirm each company sits in exactly one split | No company in two splits unless the benchmark is declared in-distribution; file the company-to-split map |
| Entities | Customer, vendor, product or employee identities and their histories | Resolve entities across records (normalized names, tax ids, account ids) and intersect test entities with training entities | Overlap at or below a threshold set before the audit; file the entity overlap report and the matching rules used |
| Task families and templates | A task pattern seen in training solves its siblings in test | Cluster task prompts by template id or structure and assign whole clusters to one split | Every template cluster in one split; file the cluster list with sizes |
| Temporal boundary | Later facts, policy versions or outcomes visible to training | Compare the latest timestamp in training data with the earliest timestamp in test data, per source | Training ends before test begins, with a gap long enough for outcomes to mature; file both dates per source |
| Near-duplicates and paraphrases | Reworded or lightly edited copies of test items | Exact substring and MinHash style checks, then embedding similarity with human or model review of top matches | No confirmed duplicate pairs across splits; file thresholds, flagged pairs and decisions |
| Environment state | A database snapshot, file system or log that already contains the answer | Diff the starting state for each task against the expected end state and search the start state for answer values | Answer values absent from the start state; file the state diff for a sample of tasks |
| Access history | People or pipelines that saw test items while tuning prompts or training | Review access logs, prompt repositories and few-shot pools for test item ids | No test items in prompts or tuning sets; file the access review and named owner |
| Public exposure | Test items or their sources published on the web or in a repository | Search for distinctive strings, embed a canary marker and check whether models complete it | No public copies found; file search queries, dates and canary results |
How do you catch near-duplicates and paraphrased tasks?
Exact matching catches copies; it does not catch rewrites, so you need both. Lee and colleagues showed that common language-model datasets contain many near-duplicates, that train-test overlap affected more than 4 percent of the validation sets of the standard datasets they studied, and that exact substring matching plus hash-based approximate document matching can find and remove much of it.
Rewording defeats those filters. Yang and colleagues showed that paraphrased or translated benchmark items pass standard n-gram decontamination, that a small model trained on rephrased test items could reach scores comparable to much larger models on well-known benchmarks, and they argued that rephrased samples should be counted as contamination. Their method pairs embedding search with a language model that judges whether a flagged pair is the same item.
For enterprise records, set the thresholds per data type before you look at results. Short structured items such as invoice lines collide often by chance, so require an entity or document match as well as text similarity. Long free-text items such as ticket threads can share boilerplate signatures and macros, so strip those before comparing. Record every flagged pair and the decision made about it; an audit that only reports a final count cannot be checked.
Could a training example, attachment or derivative reveal the evaluation answer?
Yes, and in enterprise data this is the most common hidden leak. The answer to a test task often lives in a record that looks unrelated to the task: a later email in the same thread, an attachment, a summary field or a status that someone filled in after the outcome was known.
- Reply chains that quote the resolution. A ticket used as a test prompt may be followed in training by the customer's reply that restates the fix.
- Attachments and their derivatives. A credit memo, corrected invoice or signed approval form attached to a training record can state the answer to a test task about the original document.
- Derived and back-filled fields. Resolution codes, root-cause categories and approval flags are often entered after the fact; if the test asks the agent to infer them, they must not appear in the test input or the environment.
- Summaries and notes written later. Case summaries, handover notes and month-end reports compress the outcome into a sentence that a model can recall.
- Knowledge base articles written from the same incidents. If an article was drafted from a test case, the article in training data teaches the answer.
- Few-shot examples and prompt libraries. Teams often seed prompts with real solved cases; if those cases also sit in the test set, the prompt itself leaks.
- Evaluation scripts and graders. Expected outputs stored next to the environment, or graders that print the target on failure, can leak into agent traces that later become training data.
Illustrative worked example: auditing an accounts-payable agent benchmark
This example is illustrative and its details are invented to show the method; it reports no measured results. Suppose a team builds an accounts-payable exception benchmark from the records of several companies, with tasks such as deciding whether a blocked invoice should be released, short-paid or returned to the vendor. The first version splits tasks at random.
The audit finds three problems. Most vendors appear on both sides of the split, so an agent can recall each vendor's usual resolution. Many tasks come from a handful of templates, so test items are near-siblings of training items. And the environment includes the posted payment table, which shows how each invoice was eventually paid.
The team rebuilds the benchmark with three held-out views and scores each separately, so the gap between views becomes the evidence of whether the original score reflected memory or ability.
| Evaluation view | Split rule | What it tests | How to read a large drop versus the random split |
|---|---|---|---|
| Random split (original) | Tasks shuffled across splits | Performance on familiar vendors, templates and periods | Baseline only; not a claim about new work |
| Held-out vendors | Every vendor in exactly one split | Transfer to counterparties the model has not seen | The original score relied on vendor-specific recall |
| Held-out templates | Whole template clusters assigned to one split | Handling of task structures not seen in training | The original score relied on pattern matching within templates |
| Held-out period | All test tasks dated after the training cutoff, with the payment table frozen at task start | Use on next quarter's records without future outcomes in view | The original score relied on later facts or outcomes in the environment |
The exception: when overlap is what you want to measure
Overlap is not always a defect; it is a defect when the benchmark claims something the overlap does not support. A model deployed for one company on that company's own records will see the same vendors, customers and templates in production. Testing it on familiar entities is legitimate, because recall of that company's history is part of the job.
The fix in that case is labeling, not removal. Declare the benchmark in-distribution, report it separately from any held-out score and never present it as evidence of transfer to new companies. Keep the temporal boundary even here: knowing a vendor is fair, knowing how next month's invoice was resolved is not.
The recommendation has limits. A clean split shows that a score does not depend on the leaks you checked for; it cannot prove the absence of leaks you did not think of, and it cannot tell you what a model saw in pretraining. Treat the audit as evidence about specific routes, documented well enough that someone else can rerun it.
When can no clean held-out view be built, and who should review it?
Stop and narrow the scope when the audit shows you cannot build a clean held-out view of the claim you want to make. Bring in specialist review when the remaining risk is about rights, privacy or statistics rather than engineering.
At a glance
When can no clean held-out view be built, and who should review it
- Stop when every company or entity of interest is already in training data and no fresh source is available; publish an in-distribution result instead of a transfer claim.
- Narrow the scope when one axis cannot be cleaned, for example when templates are shared by every task; report the clean views you can support and name the one you cannot.
- Request a statistician when the held-out view is small, because a few dozen tasks per view leave wide uncertainty around any difference.
- Request privacy and contract review when entity resolution requires joining personal identifiers or when records come from a supplier under license terms that limit evaluation use.
- Retire or refresh test items when a public copy, canary completion or access-log hit shows exposure; do not keep reporting a number you know is contaminated.
What public benchmarks teach about refresh and retirement
Public benchmarks show that contamination is a lifecycle problem rather than a one-time check. OpenAI explained in a published post that it stopped reporting SWE-bench Verified partly because it judged the benchmark increasingly contaminated, pointing to flawed tests and to training leakage, meaning models had likely seen benchmark material before being tested.
Two defenses recur. LiveBench draws questions from recently released sources and adds or updates questions on a regular schedule, so older material rotates out before it can saturate training corpora. BIG-bench places a canary string in every task file so that the tasks can be detected and filtered if they leak into web-scraped training data. Private enterprise benchmarks can use both ideas: refresh items from newer periods and plant markers that reveal exposure.
Sourcing held-out records for a cleaner benchmark
Most leakage problems in enterprise benchmarks come from having too few independent sources, so the same companies and templates end up everywhere. Records from organizations that have never contributed to training data are the simplest way to build a held-out view that answers the transfer question.
SourceX can look for such records at U.S. businesses on a per-request basis rather than from a prebuilt catalog, and its request form spells out scope, volume, language, format, timing and licensing needs. A contributing company signs off on scope and terms before any record ships. If you need never-published records from new companies, entities or periods for a held-out benchmark, describe the split you need when you request data.
Frequently asked questions
What is an enterprise benchmark leakage audit?
It is a documented check that test items in a benchmark built from business records do not share files, organizations, entities, task templates, time periods or environment state with anything used for training or prompt tuning. Each check has a pass rule and filed evidence so another team can rerun it.
Is benchmark contamination the same as data leakage?
Contamination usually means test items or close copies ended up in a model's training data. Leakage is the broader term for any information that reaches the model or its developers before the test and would not be available in real use, including shared entities, later dates and answers sitting in the environment.
How do I split business records so the test set is clean?
Pick the unit that matches the claim: company for transfer to new customers, entity for new counterparties, and time for future use. Assign whole groups to one split, keep every test item later than the training cutoff, and report each held-out view separately.
Do canary strings stop leakage?
No. A canary marker helps detect and filter test material that has been copied into training corpora, but it does nothing about shared entities, templates or environment state, and it only helps if the people building training data look for it.
How often should a private benchmark be re-audited?
Re-run the audit whenever test items, environments, prompt libraries or the training data change, and refresh items from newer periods on a regular schedule. Retire items as soon as you find evidence of exposure.
Sources
- Kapoor and Narayanan found data leakage in 17 scientific fields that use machine learning and grouped it into a taxonomy of eight leakage types. Source
- The scikit-learn documentation explains that grouped data breaks the independence assumption, that validation groups must be absent from training, and that time-dependent data should be evaluated on future observations. Source
- Lee et al. found that train-test overlap affects over 4 percent of the validation sets of standard datasets and used exact substring and hash-based approximate matching to find near-duplicates. Source
- Yang et al. showed that rephrased benchmark samples bypass n-gram decontamination and argued that rephrased samples should be treated as contamination. Source
- LiveBench limits contamination by drawing questions from recently released sources and updating questions on a regular schedule. Source
- OpenAI said it stopped evaluating SWE-bench Verified partly because the benchmark had become increasingly contaminated, citing flawed tests and training leakage. Source
- The BIG-bench repository says every task file contains a canary string intended to keep benchmark tasks out of web-scraped training data. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.