AI uses for records
Retrieval corpus evaluation leakage: when the RAG index holds the answers
By SourceX Editorial · Updated
9 sections
Short answer
To detect retrieval corpus evaluation leakage, tag every indexed document by lineage and look for answer keys that are not intended evidence: the evaluation file, resolved copies of the cases, documents written after the question, prior model answers and reviewer notes. Search for canary strings, scan for exact and near duplicates of each question and answer, and probe the retriever with every evaluation question.
Evaluation split
Retrieval corpus evaluation
Key takeaways
- Answers in a RAG corpus are expected; answer keys that exist only because of the evaluation are leakage.
- Most leaks are ordinary: the evaluation spreadsheet, resolved duplicates, later knowledge base articles and old run logs.
- Canary strings, duplicate scans and a retrieval probe each catch a different kind of leak.
- Entity holdouts and a time boundary set before indexing prevent most leaks upstream.
- Report the corpus snapshot version and the scan limits next to every score.
How do you detect evaluation answers accidentally included in a RAG corpus?
Classify every corpus document by lineage, then hunt for answer-bearing artifacts that were not meant to be evidence: the evaluation file itself, resolved copies of the evaluation cases, documents written after the question was asked, prior model answers and annotator notes. Three scans find most of them: a canary string search, an exact and near-duplicate match of each question and answer against the indexed chunks, and a retrieval probe that runs every evaluation question and inspects the top results by lineage.
The goal is not an index free of answers. A RAG system is supposed to find the answer in the corpus. The goal is a corpus in which the answer can only be reached through the intended evidence, the way a user would reach it in production, and not through an answer key that leaked in.
The asset on this page is a retrieval corpus exclusion and audit plan: artifact classes, how each one gets in, an exclusion rule, a detection check and the legitimate exception.
Where is the line between evidence documents and answer keys?
Every document in a RAG corpus is either intended evidence or something else. Intended evidence is what a production user's question would legitimately draw on: the policy, the manual, the runbook, the contract that existed when the question was asked. An answer key is any document that states the expected answer because the evaluation case exists, or because the case was resolved after the question point.
The distinction matters because leakage in RAG differs from training contamination. Sainz and colleagues describe the worst case for language models as training on a benchmark's test split and then evaluating on it, and argue that contamination should be measured for each benchmark. In RAG, the same effect can happen at inference with no training at all: if the answer key sits in the index, the retriever can fetch it and the generator can copy it.
Open-domain question answering already showed how common overlap is. Lewis, Stenetorp and Riedel found that 58 to 71 percent of test answers in popular datasets also appeared somewhere in the training sets, and roughly 28 to 34 percent of test questions had paraphrases there. Business corpora built from tickets, email and shared drives are at least as prone to this, because the same case is copied into many systems.
Where do answer-bearing artifacts come from in business corpora?
Most leaks are not exotic. They are the ordinary byproducts of building an evaluation set from the same systems that feed the index.
- The evaluation file: a spreadsheet of questions and gold answers saved to a shared drive that the corpus crawler indexes.
- Resolved copies: the evaluation question comes from an open ticket, and the ticket's final resolution note, or a duplicate ticket for the same incident, is in the index.
- Post-question derivatives: a knowledge base article, FAQ or Slack summary written after the case, often quoting its resolution.
- Prior model answers: chat logs or cached responses from earlier evaluation runs, saved where the crawler can see them.
- Annotator and reviewer notes: comments, adjudication sheets and gold rationales stored next to the source documents.
- Attachments and exports: a PDF or export of a case thread that contains the answer, indexed separately from the thread itself.
- Same-entity siblings: other cases for the same customer, account or asset that state the outcome the question asks about.
What does a retrieval corpus exclusion and audit plan contain?
Use the table as the exclusion policy and the list as the audit procedure. Write both down with the corpus snapshot version, so the evaluation result can be tied to the exact corpus it ran against.
Every exclusion rule is defined on lineage fields: source system, document type, creation and modification time, entity identifiers and the path the crawler took. If those fields are missing, the audit cannot be completed and the result should say so.
- Freeze the corpus snapshot and record its version, source systems and crawl date.
- Add a unique canary string to every evaluation file, annotation template and run log before any crawl.
- Tag each evaluation case with its entity ids, case ids and question timestamp.
- Apply the exclusion rules and record how many chunks each rule removed.
- Run the canary search, the exact and near-duplicate scan, and the entity and time joins over the remaining chunks.
- Run the retrieval probe: query the index with every evaluation question and label the top results by lineage class.
- Review every hit that is not intended evidence; remove it or document why it stays.
- Score the system on the cleaned snapshot and report the scan limits next to the result.
| Artifact class | Exclusion rule | Detection check | Legitimate exception |
|---|---|---|---|
| Evaluation files and gold answers | Never indexed; stored outside crawled paths | Canary string search over all chunks; filename and path scan | None |
| Resolved copies of evaluation cases | Exclude the case thread and its duplicates by case and incident id | Join on case id and incident id; near-duplicate match on question text | Duplicate detection tasks, where finding the prior case is the job |
| Post-question derivatives | Exclude documents created after the question timestamp that cite the case | Time boundary filter plus citation or quote match to the case | Documents a production user would have had at question time |
| Prior model answers and run logs | Never indexed; logs written to a separate store | Exact match of stored model outputs against chunks | None |
| Annotator and reviewer notes | Excluded by folder, tag or document type | Path and metadata scan; canary in annotation templates | None |
| Same-entity siblings | Exclude records sharing the case's entity id where they state the outcome | Entity id join; retrieval probe results grouped by entity | Entity history that existed before the question point |
| Near duplicates of question and answer text | Exclude chunks over a set similarity threshold to a question-answer pair | Exact substring and near-duplicate scan of question and answer text | Boilerplate shared by all documents, such as signatures |
Which scans find leaked answers, and what does each one miss?
Canary strings are the cheapest check. BIG-bench places a canary string in every task file so that benchmark tasks can be identified if they leak into web-scraped training data. The same trick works inside an organization: a unique string in every evaluation file and template turns a leak into a simple text search. It misses copies made by retyping or summarizing.
Exact and near-duplicate scans catch copies. The ExactSubstr tool released with Lee and colleagues' deduplication work finds repeated substrings with suffix arrays and used a 50-token threshold in their paper, and the same release includes NearDup clusters for near-duplicate documents. Run question and answer text against the chunk store at a threshold suited to your documents. These scans miss paraphrases and answers that are stated in different words.
The retrieval probe catches what the system would actually see. Query the index with each evaluation question, take the same top results the generator would get, and label them by lineage. It is the only check that measures exposure through the real retriever, but it only finds what that retriever ranks highly today, so rerun it when the embedding model or chunking changes.
Set thresholds from a labeled sample rather than by guess. Take a few dozen evaluation questions, label their top retrieved chunks by hand as evidence or answer key, and choose the similarity threshold that flags the answer keys without flagging shared boilerplate. Record the threshold, the sample and the miss rate on that sample in the audit record, because a reader of the final score needs to know how hard the corpus was searched.
How should entity, time and task family splits be set before indexing?
The cleanest fix is upstream: decide which entities, time periods and task families belong to evaluation before building the corpus, and keep the corpus on the other side of that line.
Group splitting is standard practice in model validation. Scikit-learn's GroupKFold, for example, ensures that the same group is not represented in both the testing and the training set, and TimeSeriesSplit evaluates on later observations than it trains on. The RAG equivalents are an entity holdout, in which evaluation customers or accounts contribute no records to the index beyond what existed at question time, and a time boundary, in which nothing created after the question point is indexed.
Corpus dates matter even for public data. The original RAG paper by Lewis and colleagues built its retrieval index from a December 2018 Wikipedia dump split into 100-word chunks. Recording a snapshot date like that is what makes a time boundary checkable later.
What did an audit find when a help desk assistant scored too well?
All figures in this example are illustrative. An IT team builds 250 evaluation questions from open help desk tickets and indexes its ticket system, knowledge base and shared drive. The first run answers 84% of questions correctly, well above what pilot users report.
The audit finds three leaks. The canary search hits the evaluation spreadsheet, which someone saved to an indexed folder. The case id join finds resolved duplicates for about a fifth of the questions. The retrieval probe shows knowledge base articles written after the tickets, quoting their fixes, in the top results for another group of questions.
After exclusions, the score on the cleaned snapshot falls to 61%, closer to pilot feedback. The team keeps one exception: for questions about recurring known issues, the older knowledge base article predates the ticket and is intended evidence, so it stays in the index and is labeled as such in the audit record.
The audit record the team publishes with the score is short: the snapshot version and crawl date, chunk counts removed by each exclusion rule, canary and duplicate hits with their resolution, the probe sample size and threshold, and the share of corpus documents with incomplete lineage. A later reader can rerun every step from it.
Can answers leak around the index, through fine-tuning sets or attachments?
Yes, and the corpus is not the only route. If the generator or retriever was fine-tuned on records from the same systems, the answer can sit in model weights instead of the index. The same lineage tags used for corpus exclusion should be applied to any fine-tuning set, with the same entity and time boundaries.
Attachments deserve a separate pass because they are often indexed as their own documents. A PDF export of a resolved ticket thread is a resolved copy even if the thread itself was excluded.
Stop and narrow the evaluation when lineage fields are missing for a large share of the corpus, when access history for the evaluation files cannot be reconstructed, or when exclusions would remove documents a production user legitimately relies on. Those cases need a dataset engineer and the evaluation lead to agree a narrower claim, such as reporting results only for task families with complete lineage.
Which lineage fields should a records request ask for?
When requesting business records for a RAG evaluation, ask for the fields the audit depends on: source system, document type, creation and modification timestamps, case and entity identifiers, and thread or duplicate links. Ask that evaluation candidates and corpus documents be delivered as separate sets with the split rule stated.
Each SourceX dataset for a RAG team is sourced from U.S. businesses after the request arrives, not taken from stock, and the request covers scope, volume, language, format, timing and licensing needs, which leaves room to name the lineage fields and the split rule up front.
Frequently asked questions
What is retrieval corpus evaluation leakage?
It is when the documents a RAG system can retrieve include the evaluation's answers through a route the production system would not have, such as the gold answer file, a resolved copy of the case or a document written after the question. The score then measures lookup of the answer key rather than retrieval and reasoning over intended evidence.
Is it leakage if the answer is in the RAG corpus at all?
No. A RAG system is meant to find answers in its corpus. It is leakage when the answer is reachable through an artifact that exists only because of the evaluation, or that was created after the question point, instead of through the intended evidence.
What is a canary string?
A unique, meaningless string placed in evaluation files so that any copy of them can be found by a text search. BIG-bench uses one in every task file to identify tasks that leak into web-scraped training data, and the same approach works for internal corpora.
How is this different from training data contamination?
Training contamination puts evaluation items into model weights. Corpus leakage puts them into the retrieval index, where they can be fetched at inference with no training involved. Audit both when the same source systems feed the corpus and any fine-tuning set.
How often should the audit be rerun?
Whenever the corpus snapshot, crawl paths, chunking, embedding model or evaluation set changes. The retrieval probe in particular depends on the current retriever, so a result from a previous index version does not carry over.
Sources
- Lewis, Stenetorp and Riedel found that 58 to 71 percent of test answers in popular open-domain QA datasets also occur in training sets, and about 28 to 34 percent of test questions have paraphrases there. Source
- Sainz and colleagues describe training on a benchmark's test split and then evaluating on it as the worst case, and call for measuring contamination for each benchmark. Source
- BIG-bench places a canary string in every task file to prevent benchmark tasks from leaking into web-scraped training data. Source
- The deduplication release for Lee and colleagues' paper provides the ExactSubstr tool, used with a 50-token threshold in the paper, and NearDup near-duplicate clusters. Source
- Scikit-learn's GroupKFold keeps the same group out of both test and training sets, and TimeSeriesSplit evaluates on later observations. Source
- The original RAG paper built its retrieval index from a December 2018 Wikipedia dump split into disjoint 100-word chunks. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.