Pinned corpus snapshots for reproducible RAG evaluation

Why RAG scores drift when the knowledge base changes

RAG scores drift because the ground truth is defined relative to a document set, and enterprise document sets can change weekly. The WixQA authors tie their enterprise RAG benchmark to the specific knowledge-base snapshot from which the answers were derived and release that snapshot alongside the QA pairs [1]. Classic IR evaluation has long worked the same way; TREC distributes relevance judgments per collection, and qrels are only meaningful against the collection they were judged on [2].

Drift shows up in four recognizable ways:

• Silent answer change. A pricing article is edited, so the reference answer in your QA set is now wrong, and a correct system is penalized. FreshQA was built around exactly this class of questions, whose correct answers depend on when you ask [3].
• Orphaned qrels. A judged-relevant document is deleted or re-IDed in the CMS export, so recall@k drops for every system at once.
• Unjudged newcomers. New articles that answer a question well are retrieved but have no judgment, so they count as non-relevant and depress precision.
• Chunk boundary shift. The same document re-chunked with a new splitter produces different passage IDs, breaking passage-level labels even though no content changed.

None of these show up as errors. They show up as a 2-point move on a dashboard that someone will try to explain as a model effect.

What a pinned snapshot has to contain

A usable snapshot is a manifest, not a folder name like kb_final_v2. It should list every document by stable ID with a content hash, record the export time and source system, and declare the processing parameters needed to rebuild the exact index. Benchmarks such as BEIR ship corpus, queries and qrels as one package for this reason [4], and dataset documentation practice from Datasheets for Datasets asks for composition, collection process and maintenance plans to be written down [7].

Illustrative example: invented to show structure; it does not describe an available dataset.

snapshot_id: kb-support-2026-09-30-r1
parent_snapshot: kb-support-2026-06-30-r1
source_system: "Zendesk Guide export (help center articles)"
exported_at: "2026-09-30T23:59:59Z"
license_ref: "license schedule A, records list v3"
documents:
  count: 4812
  manifest_file: docs_manifest.parquet   # doc_id, version_id, sha256, locale, updated_at, status
  hash_algorithm: sha256
changes_since_parent:
  added: 211
  modified: 389
  removed: 57          # removed_reason: retired | license_withdrawal | privacy
processing:
  parser: "html-to-markdown, tables preserved"
  chunker: "heading-aware, 512 tokens, 64 overlap"
  chunk_id_rule: "{doc_id}:{version_id}:{ordinal}"
  embedding_model: "embed-model-x, revision pinned"
  index: "HNSW, M=32, efConstruction=200"
eval_bindings:
  qa_set: qa-v4 (bound to this snapshot_id)
  qrels: qrels-v4-doc and qrels-v4-passage

Three fields do most of the work. version_id distinguishes an edited article from its predecessor, sha256 lets anyone verify the bytes, and chunk_id_rule makes passage IDs deterministic so passage-level qrels survive a rebuild. If you publish the dataset internally, expressing the manifest in Croissant JSON-LD gives tooling a machine-readable description of files and record fields [8].

Freezing the index, not just the documents

Freezing documents is necessary but not sufficient; the retrieval stack between the documents and the scores must also be pinned. Two runs on the same corpus can diverge because the embedding model was silently upgraded behind an API alias, the tokenizer changed chunk lengths, or approximate nearest-neighbor search returned slightly different neighbors after a rebuild.

Practical rules for a frozen eval index:

• Pin embedding models by explicit revision or checkpoint hash, never by a floating alias.
• Store the built index artifact (FAISS, HNSW graph, or vector-database snapshot) under the snapshot ID, rather than rebuilding it per run.
• Record ANN search parameters (efSearch, nprobe, top-k) in the run config, and log them with each result.
• Keep a small exact-search (brute-force) baseline on the same snapshot so ANN nondeterminism can be separated from model effects.
• Store BM25 parameters and the analyzer configuration for lexical or hybrid retrievers.

The production index can keep moving. The eval index is a read-only artifact whose ID appears in every score you report.

Binding qrels and QA pairs to a corpus version

Every relevance label and reference answer should carry the snapshot ID it was created against, and your harness should refuse to score a run when the IDs disagree. This mirrors the TREC convention that a run is evaluated with the qrels for its collection [2]. For document-level qrels, key labels on doc_id plus version_id; for passage-level qrels, key them on the deterministic chunk ID.

When you build a new snapshot, carry labels forward mechanically: unchanged hashes inherit their judgments, modified documents are flagged for re-judgment, and removed documents drop out with a logged count per topic. Topics that lose all relevant documents should become explicit unanswerable cases or be retired, not left silently empty. For how to construct the question side, see question-answer-citation triples for RAG evaluation and testing abstention with unanswerable questions.

Re-baselining when licensed documents must be removed

When a license or privacy obligation requires removing documents, you cannot keep scoring against the old snapshot, so you create a successor snapshot and re-baseline every system you compare. Removal is the case where reproducibility and compliance pull in opposite directions: the old snapshot is what makes historical scores reproducible, but it may be exactly what you are no longer allowed to keep.

Illustrative example: invented to show structure; it does not describe an available dataset.

Step 6 is where teams get caught. Versioned storage systems keep deleted objects reachable from older commits: lakeFS retains them until garbage collection, under its retention rules, hard-deletes objects no longer referenced [5], and DVC keeps data in its cache and remote until dvc gc removes it [6]. Embedding vectors and cached chunks derived from the document need the same treatment. Retention and deletion terms are covered in caching and retention limits for licensed RAG content.

Storage choices for snapshot history

Choose storage that gives you immutable references for reproducibility and a real hard-delete path for removals. Data-versioning tools (lakeFS, DVC) and table formats with snapshot or time-travel semantics all support pinning, but each needs an explicit collection step before deleted bytes actually leave storage [5][6].

A workable pattern is to keep the current snapshot and its immediate parent online, archive older manifests (IDs and hashes only, no content), and keep historical score reports tied to manifest IDs. You lose the ability to re-run very old evaluations, but you can still prove what they were run against. For the related problem of stale and superseded versions inside a single snapshot, see knowledge-corpus quality: duplicates, stale versions and conflicts.

What to ask a content supplier before you license evaluation data

Licensed evaluation data should arrive with its corpus version already defined, so ask for the snapshot terms during diligence rather than after delivery. These are generic questions for any supplier; SourceX prepares diligence materials per dataset covering source, rights, preparation and allowed use, and delivers under a license that defines the records, uses, term and delivery. You can describe the corpus you need on the SourceX buyers page.

• Is there a document-level manifest with stable IDs, version IDs and content hashes?
• What was the export cutoff, and from which system (help center, wiki, ticketing, document management)?
• Are QA pairs, citations or qrels keyed to that exact export?
• How will updates arrive: full re-exports, or deltas with added, modified and removed lists?
• How are removals communicated, and what deletion evidence is expected from you?
• Were personal details removed or replaced before delivery, and by what recorded method? See de-identifying a RAG corpus without breaking retrieval.
• Which metadata fields ship with each document? Compare against metadata a licensed RAG corpus should ship with.

If you are assembling a full test collection rather than snapshotting an existing one, start from building a private domain retrieval test collection. For the wider buying context, see the RAG content licensing and retrieval data guide, RAG evaluation datasets from real company documents and the glossary entry on data lineage.

Sourcing a licensed corpus with a pinned evaluation snapshot

SourceX sources operational datasets, such as support histories, documents and engineering records, from US companies on request, and every release is approved by the supplying company. Each dataset is rights-reviewed and delivered under a license that defines the records, uses, term and delivery. Describe the corpus and snapshot requirements you need at sourcex.si/buyers.