AI uses for records
Retrieval hard negatives from business records: the wrong customer's file
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Retrieval hard negatives from business records work best when the negative is the same kind of document for a different customer: the same contract template, invoice layout or service report, with a different account behind it. Label it negative only when the query is about one specific customer, give every customer a consistent pseudonym before pairing, and record who could see both documents so the pair never teaches cross-customer access.
Retrieval
Retrieval hard negatives
Key takeaways
- In templated business corpora, the hardest negative is usually the same document type for a different customer.
- Label a wrong-customer document negative only for account-specific queries; for generic questions it may be a valid answer.
- Check structured keys such as customer or contract identifiers before trusting similarity scores to decide the label.
- Use consistent per-customer pseudonyms; a single generic mask turns every cross-customer pair into a false negative.
- Keep negatives inside what a role may legitimately see, and test that the system abstains when only another customer's document is retrieved.
Why is another customer's document the hardest negative in a business corpus?
Because it matches the query on almost everything except the one thing that matters. Business systems are built on templates: master service agreements, renewal quotes, invoices, inspection reports and onboarding checklists repeat the same headings and clauses for hundreds of accounts. When someone asks for the notice period in one customer's agreement, the closest passage by wording is often the identical clause in another customer's agreement.
That is the definition of a hard negative. The Dense Passage Retrieval paper built its BM25 negatives from top-ranked passages that do not contain the answer but match most question tokens, and found that adding hard negatives to in-batch negatives improved training. In a business corpus, a wrong-customer document is a stronger version of the same idea: it shares the template, the vocabulary and often most of the numbers.
A retriever that has never been trained or tested on these pairs learns to rank by template similarity. It then returns a correct-looking clause from the wrong account, and the model built on top quotes it confidently. In customer-facing workflows that is both an accuracy failure and a disclosure risk.
When is the wrong customer's document a true negative, and when is it a false one?
It is a true negative only when the query is scoped to one customer and the other document cannot answer for that customer. If the query is generic, or the clause is identical standard language that governs every account, the other customer's document may be a perfectly good answer, and labeling it negative trains the retriever to reject relevant evidence.
False negatives are not a rare edge case in retrieval data. The RocketQA authors manually examined top-ranked passages from their retriever that were not labeled positive in MS MARCO and reported that about 70 percent were actually positive or highly relevant. Business corpora with shared templates are, if anything, more prone to this. The rule below makes scope the first label, before any mining starts. Examples are illustrative.
| Query scope | Illustrative query | Other customer's matching document is | Labeling rule |
|---|---|---|---|
| Named customer, account-specific fact | What is the renewal notice period in the CUST-0417 agreement? | Hard negative | Label negative if the other agreement differs or is not the anchor customer's |
| Named customer, standard clause | Does the CUST-0417 agreement include our standard limitation of liability? | Possible positive for the clause text, negative for the account | Keep as negative only when the answer must come from the anchor's own signed copy |
| Generic policy or template question | What does our standard MSA say about renewal notice? | Positive or neutral | Never label negative; use the template itself as the positive |
| Time-scoped, same customer | What did CUST-0417's 2024 renewal quote include? | Negative, alongside the same customer's other-year quote | Mine both cross-customer and cross-period negatives |
| Aggregate or comparative | Which customers have a 60-day notice period? | Not applicable | Exclude from pairwise hard negative mining |
How should mining settings change when negatives come from other customers?
Mine by entity first and by similarity second. Instead of taking whatever a retriever ranks just below the positive, select candidate negatives that share the positive's document type and template but belong to a different customer, then use similarity scores only to choose among them and to drop suspicious near-duplicates.
Similarity thresholds still matter. The sentence-transformers mine_hard_negatives utility exposes range_min and range_max to skip the very closest matches or limit candidates, max_score to drop candidates too similar to the anchor, and absolute and relative margins to skip negatives whose score sits within a set distance of the positive pair. Those controls exist because the most similar candidates are where false negatives concentrate. For wrong-customer negatives, a candidate whose text is effectively identical to the positive should go to a person, not be dropped or kept automatically, since identical text is exactly the case where scope decides the label.
RocketQA's response to false negatives was to filter candidates with a stronger cross-encoder before using them. The business equivalent is cheaper and more reliable: check the structured fields. If the positive and the candidate share a customer identifier, contract number or account key, it is not a cross-customer negative. If they differ only in that key and the query is account-specific, it is.
Measure whether the negatives helped with a metric that names the failure. Overall recall can rise while the retriever still prefers the neighbor's document, so report a wrong-entity rate as well: the share of account-specific test queries where the top-ranked passage belongs to a different customer. Split the held-out set by query scope, keep customers in the test set out of training entirely, and compare against a retriever trained without cross-customer negatives. If the wrong-entity rate falls but generic-question recall also falls, the labeling rule is probably marking valid template answers as negatives.
Ratios matter less than coverage of document types. A corpus where every cross-customer negative comes from one contract template teaches the retriever to read one party block. Spread negatives across the template families the deployment will search, such as agreements, quotes, invoices and service reports, and record the family on each row so the balance can be audited.
What should the privacy reviewed hard negative collection specification include?
One row per query, positive and negative triple, carrying enough metadata that a privacy reviewer and a retrieval engineer can each approve it without the other present. The specification below is a starting template; field names are illustrative and should be mapped to your own corpus.
- Keep the surrogate map outside the training data and under separate access control; only the pseudonyms travel with the triples.
- Record the corpus snapshot date so a negative can be traced to the exact text that was paired.
- Version the specification itself, and note which rows change when a customer's contract is amended or a record is deleted.
| Field | What it records | Illustrative value | Reviewer who relies on it |
|---|---|---|---|
| query_scope | Account-specific, standard-clause, generic, time-scoped or aggregate | account-specific | Retrieval engineer |
| anchor_entity | Consistent pseudonym of the customer the query is about | CUST-0417 | Both |
| positive_ref | Document and version identifier that answers the query | MSA-0417 v2, section 9.1 | Retrieval engineer |
| negative_ref | Document and version identifier used as the negative | MSA-0932 v1, section 9.1 | Retrieval engineer |
| negative_type | Cross-customer same template, same customer other period, other document type | cross-customer same template | Retrieval engineer |
| distinguishing_evidence | The field or span that makes the negative wrong for this query | Notice period 60 days vs 30 days; different party | Both |
| false_negative_check | How scope and content were checked, and by whom | Key mismatch confirmed; reviewer sample | Retrieval engineer |
| permission_context | Which roles could legitimately see both documents in the source system | Account managers for the region; not customer portal users | Privacy reviewer |
| privacy_treatment | De-identification method, surrogate map version and residual risk notes | Names and addresses replaced with consistent surrogates, map v3 | Privacy reviewer |
| review_status | Approved, approved with edits, excluded, with reason and date | Approved 2026-10-18 | Both |
How do you de-identify both documents without erasing the difference?
Replace identifiers with consistent surrogates, not a single generic mask. If every customer name becomes the same placeholder, the positive and the wrong-customer negative become textually identical, and you have manufactured a false negative. Each real customer needs a stable pseudonym used in the query, the positive and every negative, so the retriever can still learn that CUST-0417 and CUST-0932 are different accounts.
Automated detection helps but does not finish the job. Presidio, an open-source identification and anonymization toolkit now maintained as a community project, states that because it uses automated detection there is no assurance it finds all sensitive information, and that additional systems and protections should be used. Templated business documents add their own traps: account numbers inside file names, signatures in scanned annexes, addresses in letterheads and customer names inside product codes.
Treat cross-customer pairs as a higher-risk release than single documents, because each training example now places two customers' records side by side. NIST SP 800-188 on de-identifying datasets recommends re-identification studies to gauge risk and a disclosure review board to oversee the process. A small internal version of both, a sample re-identification attempt on finished triples and a named approver, is proportionate for most business corpora.
Should a hard negative ever come from a document the user could not see?
A hard negative tells the retriever which document is wrong for the query; it says nothing about whether the user was allowed to see either one. Keep the two questions separate in the specification and in evaluation, because a model that learns to rank correctly across customers can still be deployed where it should never see other customers at all.
In production, access is usually enforced before ranking. Microsoft's security filter pattern for Azure AI Search, for example, stores group identifiers in a filterable field and excludes documents that do not match the caller, while noting that the filter is not itself authentication. OWASP's 2025 guidance on vector and embedding weaknesses calls for permission-aware stores and strict logical partitioning between user groups to prevent cross-context leakage. Hard negatives belong inside that boundary: they should model confusion among documents a given role may legitimately see, such as an account manager covering many customers.
Evaluation should add one more expectation: abstain. If the only document retrieved for an account-specific question belongs to another customer, the correct behavior is to say the answer is not available, not to answer from the neighbor. Agent benchmarks encode the same principle in policy; the tau-bench retail policy limits the agent to one authenticated user per conversation and requires it to deny requests about any other user.
Illustrative worked example: two agreements from one template
This example is illustrative and invented for this page. A facilities services company stores signed master service agreements for every client in a document system, all generated from the same template. Two clients, pseudonymized as CUST-0417 and CUST-0932, signed in different years; one negotiated a 60-day renewal notice period and the other kept the template's 30 days.
- Query: What notice does CUST-0417 need to give before the agreement auto-renews?
- Positive: MSA-0417 version 2, section 9.1, stating 60 days.
- Cross-customer negative: MSA-0932 version 1, section 9.1, stating 30 days, identical in every other word.
- Same-customer negative: MSA-0417 version 1, superseded by an amendment, stating 30 days.
- Distinguishing evidence: the party block and the notice period; recorded in the specification row.
- Permission context: both agreements visible to the regional account management group; neither visible to client portal users.
- Exception: the query What notice period does our standard MSA use? is generic, so MSA-0932's clause is a valid positive for it and must not be reused as a negative there.
Which records should never be paired across customers?
Some records are too sensitive or too restricted to place side by side, even after de-identification. Exclude them from cross-customer mining and, if they are needed at all, use same-customer negatives only.
- Records under a confidentiality clause that bars use beyond servicing that customer, until the company confirms the license covers it.
- Health, financial account, payment card, government identifier or children's data, where pairing raises re-identification risk sharply.
- Small-population accounts where the template differences alone identify the customer, such as the only client in a region or industry.
- Documents whose permission metadata is missing or inconsistent, since the permission_context field cannot be filled honestly.
- Records subject to a pending deletion request or retention expiry, which would have to be removed from every triple that used them.
Where do licensed wrong-customer pairs come from?
Wrong-customer negatives exist only in real multi-customer archives, which is why they are hard to synthesize convincingly. Using them requires more than system access: the documents belong to the business, the people named in them have privacy interests, and the right to use them for training or evaluation comes from an agreement with the company, not from de-identification alone.
SourceX does not keep multi-customer archives in stock; it sources them on request from U.S. businesses and handles the rights review, contract, preparation and delivery with the company that owns them. Retrieval teams spell out the document types, template families, volume and format they need on the request form.
Frequently asked questions
How do you construct retrieval negatives from the wrong customer's documents?
Start from account-specific queries, pick candidate documents of the same type and template that belong to a different customer, and confirm the difference with structured fields such as the customer or contract key. Pseudonymize every customer consistently, record who could see both documents, and send near-identical pairs to a person for review.
Are wrong-customer documents always good hard negatives?
No. They are negatives only for queries scoped to one customer. For generic questions about a standard template or policy, another customer's copy of the same clause can be a valid answer, and labeling it negative creates a false negative.
Why not redact all customer names before mining negatives?
Because a single generic mask makes the positive and the negative identical, so the retriever cannot learn the difference. Consistent surrogates keep each customer distinct while removing the real identity. The surrogate map should be stored separately from the training data.
Do cross-customer hard negatives create a data leakage risk?
They can if they teach a model to retrieve across boundaries it should not cross. Keep negatives within documents a given role may legitimately see, enforce permissions before ranking in production, and test that the system abstains when only another customer's document is available.
What review does a hard negative collection need before use?
A retrieval engineer should confirm scope and false-negative checks, and a privacy reviewer should approve the de-identification, the surrogate map handling and the permission context. Counsel should confirm that the license from the business covers training or evaluation use of the paired records.
Sources
- Dense Passage Retrieval trains with in-batch negatives plus BM25 negatives, defined as top BM25 passages that do not contain the answer but match most question tokens. Source
- RocketQA reports that about 70 percent of top-ranked MS MARCO passages not labeled positive were actually positive or highly relevant, and filters hard negatives with a cross-encoder. Source
- The sentence-transformers mine_hard_negatives utility provides rank range, maximum score and absolute or relative margin settings to skip candidates too close to the anchor or positive. Source
- Presidio identifies and anonymizes sensitive entities in text and images, and states that automated detection cannot be relied on to find all sensitive information. Source
- NIST SP 800-188 recommends re-identification studies to gauge de-identification risk and a disclosure review board to oversee the process. Source
- Azure AI Search's security filter pattern stores group identifiers in a filterable field to exclude documents a caller should not see, and notes the filter is not authentication. Source
- OWASP LLM08:2025 warns of cross-context leaks in shared vector databases and recommends permission-aware stores with strict logical partitioning between user groups. Source
- The tau-bench retail agent policy limits the agent to one user per conversation and requires it to deny requests related to any other user. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.