AI uses for records
Multi-tenant benchmark leakage: one customer in two companies' records
By SourceX Editorial · Updated
10 sections
Short answer
When the same customer appears in two suppliers' records, resolve it to one entity, link every record to its supplier and to each entity it mentions, and split by the connected groups that result instead of by supplier. Whole groups go to training or evaluation. Records that cannot follow the rule go into a split exception log with a leak test, a mitigation and a separate reporting slice.
Evaluation split
Multi tenant benchmark
Key takeaways
- Holding out a supplier does not hold out its customers; a shared customer's history in training can answer evaluation cases.
- Group splitters take one label per record, so combine tenant and entity links into connected components and split on those.
- Match in layers from strong identifiers to probabilistic linkage to text similarity, and record which layer made each link.
- Attachments, later records, derived fields and shared indexes leak answers that a scan of the main text misses.
- Log every override in a short exception log and report affected tasks as a separate slice.
- When one entity links most records, switch to masking, a time split or a documented exception rather than a group split.
How do you handle the same customer appearing in two suppliers' records?
Resolve the customer to one entity across both suppliers, then split by connected groups rather than by supplier. Link every record to its supplier tenant and to each resolved customer entity it mentions, compute the connected components of that graph, and assign whole components to either training or evaluation. Records that cannot follow the rule go into a split exception log with a reason, an owner and a test for whether they leak an answer.
The reason is simple. A held-out supplier is only held out if nothing about its evaluation cases is visible elsewhere in training. When a property management firm, a hospital group or a national retailer buys from both suppliers, the training side can contain that customer's addresses, contract terms, dispute history and outcomes, and a model scored on the other supplier's records may be recalling the customer rather than doing the task.
This page is for evaluation leads and dataset engineers assembling benchmarks from several businesses' operational records, which is the multi tenant benchmark leakage problem in its plainest form.
Why isn't a supplier-level split enough?
Because the supplier is only one of the things records share. scikit-learn's cross-validation guide states the general rule: when the process that generates data yields groups of dependent samples, the usual independence assumption is broken, and to measure generalization to unseen groups every sample in the validation fold must come from groups not represented at all in the paired training fold. Its group splitters take one group label per sample. A record that belongs to supplier B and also mentions customer X belongs to two groups, so neither label alone keeps it apart from customer X's records at supplier A.
Overlap of this kind is not a corner case in published benchmarks. Lewis, Stenetorp and Riedel found that in three popular open-domain question answering datasets 60 to 70 percent of test-time answers also appear in the training data, and that 28 to 34 percent of test questions have a near-duplicate paraphrase in training. Lee and colleagues found train-test overlap affecting over 4 percent of the validation sets of standard language modeling datasets. Kapoor and Narayanan's survey found leakage problems in at least 294 papers across 17 fields.
Business records add a twist those studies did not face: the shared thing is often a real organization or person, not a repeated sentence. Text deduplication, such as the exact-substring tool Lee and colleagues released with their paper, is built to find sequences repeated across a corpus. It will not see a shared customer, because the two suppliers describe that customer in different words, systems and formats.
Which split unit blocks which leak?
Pick the unit by asking what a model could memorize that would answer an evaluation case. No single unit blocks everything, so most multi-supplier benchmarks combine two: the customer entity group plus a time boundary, with task family as a third check for templated work.
| Split unit | Leak it blocks | Leak it misses | Use when |
|---|---|---|---|
| Supplier tenant | Supplier-specific templates, staff names, house style | Shared customers, shared vendors, industry-wide templates | Testing transfer to a new company |
| Customer entity group | Recall of a customer's history, terms and outcomes | Templated tasks that differ only in the customer | Customers recur across suppliers or years |
| Time boundary | Later records that state the outcome of an earlier case | Long-lived facts stated before the boundary | Tasks predict or decide something that resolves later |
| Task family | Near-identical tasks generated from one template or playbook | Entity recall across different task types | Tasks were authored or derived in batches |
| Connected component of all of the above | Most of the leaks listed here | Paraphrased knowledge with no shared key | Default for multi-supplier benchmarks, unless it collapses |
How do you find shared customers without over-matching?
Match in layers, from strong identifiers to weak evidence, and keep the layer that produced each link. Strong keys come first: tax or registry numbers, email domains, and for legal entities the Legal Entity Identifier, which GLEIF describes as a 20-character alphanumeric code based on the ISO 17442 standard for uniquely identifying legal entities. Many small business customers will not have one, so treat it as a bonus key rather than a requirement.
Probabilistic linkage handles the rest. Splink, an open-source package for probabilistic record linkage based on the Fellegi-Sunter model, links records that lack a shared identifier and clusters the pairwise links into an estimated entity id. Its documentation notes that it performs best with several columns that are not highly correlated and is not designed for a single bag-of-words column such as company name alone, which is a fair warning for B2B records where the name is often all you have.
Text similarity catches the residue: the same quote, contract clause or email thread copied into both suppliers' systems. MinHash with locality-sensitive hashing finds sets above a Jaccard similarity threshold quickly, but the datasketch documentation is explicit that the method returns false positives and misses some qualifying pairs. Record the threshold and treat it as one of the scan's limits.
Run matching inside a controlled environment on normalized or hashed keys, and keep raw identifiers out of the evaluation team's working copy. The goal is a group label, not a merged customer file.
The entity overlap review
The review is a register with one row per resolved entity that appears in more than one tenant, or in both splits after a naive split. It is the evidence that the final split was checked, and it is what a reviewer reads when a score looks too good.
At a glance
The entity overlap review
| Field | What to record | Illustrative entry |
|---|---|---|
| Entity group id | Stable id for the resolved customer, independent of any supplier's own id | ENT-0412 |
| Tenants and record counts | Each supplier tenant holding records about the entity, with counts | Supplier A: 63 records; Supplier B: 18 records |
| Match evidence and layer | Which key or model linked the records and its score | Shared email domain plus Splink match above the set threshold |
| Entity type | Organization, person, location or asset; persons get stricter handling | Organization (property manager) |
| Task families touched | Which evaluation task types use this entity's records | Invoice dispute resolution; service scheduling |
| Time span | First and last record dates per tenant | Supplier A 2021 to 2025; Supplier B 2024 to 2025 |
| Answer exposure | Whether any training record states or implies an evaluation answer | Yes: credit memo outcome visible in Supplier A notes |
| Decision | Whole group to train, whole group to eval, exclude, or exception | Whole group to eval; Supplier A records removed from train |
| Reviewer and date | Who decided and when, for audit | Dataset engineer, 2026-10-18 |
The split exception log
Some records cannot follow the group rule without destroying the benchmark, and those decisions should be written down rather than made silently in code. The exception log is short by design: if it grows past a handful of rows per thousand records, the split rule is wrong, not the data.
Check that the split did its job by scoring the same model on three slices: evaluation cases with no shared entity, cases covered by an exception, and, as a diagnostic only, cases a naive supplier split would have leaked. A clear gap between the first and the last tells you how much the overlap was worth to the model and whether the exceptions are behaving more like the clean slice or the leaked one.
| Field | What to record | Illustrative entry |
|---|---|---|
| Exception id and scope | Which records or group the exception covers | EXC-07: ENT-0090 records in both splits |
| Rule broken | Which split rule the exception overrides | Customer entity group kept whole |
| Reason | Why the rule cannot be followed | Entity links a large share of all records; keeping it whole empties the training set |
| Leak test performed | The check showing no evaluation answer is exposed | Eval tasks touching ENT-0090 ask about events after the time boundary only |
| Mitigation | What was changed to limit the leak | Entity name and identifiers masked in train; outcome fields removed |
| Reporting rule | How scores on affected tasks are shown | Reported separately as an overlap slice |
| Owner and expiry | Who owns the exception and when it is reviewed again | Evaluation lead; next dataset release |
Where do attachments, later notes and derived fields leak the answer?
They hide in places a scan of the main text field never reads, and those channels are where most real leaks in business records sit. Check each one for every group the review marks as shared:
At a glance
Where do attachments, later notes and derived fields leak the answer
- Attachments and forwarded threads: a PDF invoice or a forwarded email from supplier B pasted into supplier A's ticket.
- Signatures and boilerplate: a customer's address, account number or contract reference repeated in every message.
- Later records: a follow-up note, credit memo or closure code that states how the earlier case ended.
- Derived fields: summaries, tags, sentiment scores and resolution codes produced after the outcome was known.
- Shared vendor templates: two suppliers running the same SaaS product generate near-identical workflow text, which links task families across tenants even when no customer is shared.
- Embeddings and indexes: a retrieval index built over all tenants before the split will serve evaluation documents at training time.
- Access history: anyone who viewed the evaluation set while building prompts, rubrics or synthetic data is a possible channel; log who had access and when.
Worked example: two HVAC distributors and one property manager
Illustrative case. Two HVAC parts distributors supply records for an invoice-dispute benchmark: 4,000 dispute cases from Supplier A and 1,500 from Supplier B. The team first holds out Supplier B as the evaluation tenant. Entity resolution then finds 37 customers present at both suppliers, and one of them, a regional property manager, appears in 210 Supplier A cases and 44 Supplier B cases.
The answer-exposure check shows the problem. Supplier A's notes record the property manager's standing agreement that freight charges on emergency orders are waived, which is exactly the fact that decides 19 of the 44 Supplier B evaluation disputes. A model trained on Supplier A would score well on those cases by recalling the customer, not by reading the dispute.
The fix moves 36 of the 37 shared groups into whichever side holds most of their records and removes their records from the other side, at a cost of about 6 percent of training cases (illustrative). The property manager is too large to move without unbalancing the split, so it becomes exception EXC-01: its Supplier A records stay in training with outcome fields masked, and its 44 evaluation cases are scored as a separate slice. Reporting the slice next to the clean set shows whether the masked training records still help.
When does the overlap problem need a different design?
Stop and redesign when the connected components collapse. In concentrated industries one large customer, payer or platform can link most records into a single component, and a group split then has nothing to split. The usual options are to drop that entity from evaluation, switch to a time-based split with the entity masked, or accept the overlap and report it as a known limit with a separate slice.
Narrow the scope when the matching itself is weak. If the only shared field is a free-text company name, probabilistic linkage will over-match or under-match and the review will give false comfort. Request a specialist privacy review before matching on personal data such as patient, employee or consumer identifiers, because linking two companies' records about the same person can create a combined profile neither company holds today. That is a legal question for counsel, not a dataset engineering choice.
Write the scan's limits into the benchmark card: which identifiers were used, the similarity threshold, the date of the scan, and the record types that were not checked. A future reader can then judge whether a surprising score might be overlap rather than skill.
Sourcing multi-supplier evaluation data with overlap in mind
Benchmarks built from several businesses are worth more than single-company sets precisely because they test transfer, and that value depends on knowing where the businesses overlap. When requesting records, ask for stable customer keys or hashed identifiers, record dates and attachment manifests so the review above can be run before the split is fixed.
Multi-supplier requests suit the way SourceX works, since it assembles each dataset from U.S. businesses after a request arrives rather than from stock; the request-data form takes scope, volume, language, format, timing and licensing needs, and split and overlap requirements can be written in alongside them. Every contributing business signs its own agreement on scope before delivering anything, and each source goes through its own rights, privacy, quality and buyer-fit review.
Frequently asked questions
What is multi tenant benchmark leakage?
It is evaluation leakage that arises when a benchmark combines records from several organizations and something shared across them, usually a customer, vendor or template, lets a model trained on one tenant's records answer another tenant's evaluation cases. It makes scores look like transfer when they partly reflect recall.
Is deduplication enough to prevent it?
No. Deduplication finds repeated or near-identical text. Two suppliers usually describe the same customer in different words and systems, so the leak is an entity, not a duplicate string, and it needs entity resolution and group splits.
Can I use GroupKFold with two kinds of groups?
Not directly, because group splitters take one label per sample. Build a graph linking each record to its tenant and its resolved entities, take connected components, and pass the component id as the group label.
What if one customer appears in almost every record?
Then group splitting collapses. Mask or exclude that customer, use a time-based split, or keep it as a logged exception and report its tasks as a separate slice so readers can see its effect.
Who should review the entity overlap log?
A dataset engineer owns the matching and the evaluation lead owns the split decisions. If the matching uses personal identifiers, a privacy reviewer or counsel should approve the approach before records from different companies are linked.
Sources
- scikit-learn's cross-validation guide explains that groups of dependent samples break the independence assumption, and that measuring generalization to unseen groups requires validation samples from groups absent from the paired training fold, set through a groups parameter. Source
- Lewis, Stenetorp and Riedel (EACL 2021) found that 60-70% of test-time answers in three open-domain QA datasets also appear in training, and that 28-34% of test questions have paraphrased near-duplicates in training. Source
- Lee et al. (ACL 2022) report that train-test overlap affects over 4% of the validation set of standard language modeling datasets, and that deduplication makes evaluation more accurate. Source
- The deduplicate-text-datasets repository releases the exact-substring deduplication code behind Lee et al. for finding sequences repeated across a text dataset. Source
- Kapoor and Narayanan (Patterns, 2023) found data leakage affecting at least 294 papers across 17 fields and propose a taxonomy of eight leakage types. Source
- Splink is an open-source probabilistic record linkage package based on the Fellegi-Sunter model that clusters pairwise links into estimated entity ids; it works best with several uncorrelated columns and is not designed for a single company-name column. Source
- The datasketch documentation explains that MinHash LSH finds sets above a Jaccard similarity threshold in sub-linear time but returns some false positives and misses some qualifying sets. Source
- GLEIF describes the Legal Entity Identifier as a 20-character alphanumeric code based on ISO 17442 that enables unique identification of legal entities. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.