De-identifying a RAG corpus without breaking retrieval

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

This is the retrieval-specific slice of the problem. The full method for company data, from detection to sign-off, lives in the de-identification playbook for company data, and the wider buying context sits in the RAG content licensing and retrieval data guide.

Why blanket redaction breaks RAG retrieval

Blanket redaction breaks retrieval because it erases the tokens that link documents to each other and to queries. A guide on building evaluation sets warns that replacing every entity with a single REDACTED token can destroy coreference, formatting or retrieval behavior, and recommends anonymizing without removing the property you are testing [6]. In a RAG corpus, that property is usually "these five documents are about the same account."

The failure modes are concrete and predictable:

• Multi-document questions go unanswerable. "What did we promise the customer who escalated ticket 88213 in the renewal call?" needs the ticket, the call notes and the contract to share an entity. With redaction, all three say "[REDACTED]" and the join is gone.
• Lexical retrieval loses exact matches. BM25 and other sparse retrievers score on shared terms. When account names become one shared token, that token appears in thousands of chunks and carries almost no weight.
• Dense embeddings drift toward each other. Chunks that differed mainly by who they concerned become near-identical vectors, so top-k results fill with interchangeable chunks and near-duplicate filters may merge distinct records.
• Metadata filters stop working. If account_id or customer_name fields are nulled rather than remapped, tenant filters, access-control filters and per-entity facets break.
• Generation hallucinates bindings. When the context says "[NAME] told [NAME] that [NAME] approved it," the model guesses who did what.

Consistent, typed pseudonyms: the core design choice

The fix is pseudonymization that is consistent across the whole corpus and typed by entity class. One mapping table assigns each real entity a stable surrogate, and every document, attachment, metadata field and evaluation query is rewritten through that same table. A person keeps a person-shaped surrogate, an email keeps an email shape, and an account number keeps its length and check-digit format, so parsers, chunkers and regex-based tools still behave.

Two surrogate styles are common. Tagged surrogates ("PERSON_0412", "ORG_0077") are easy to audit and impossible to mistake for real data, but they read unnaturally and can bias generation. Realistic surrogates ("Dana Whitfield", "dana.whitfield@example.invalid") read naturally and keep tokenization closer to the original, but they can collide with real people, so use reserved domains and a surname list checked against your keep-list.

How you generate surrogates matters for compliance as well as utility. A keyed hash (HMAC) of the real name gives consistency without a lookup table, but it is derived from the identifier itself. HIPAA's de-identification rule allows a re-identification code only if it is not derived from information about the individual and the re-identification mechanism is not disclosed [2], and OCR's guidance discusses that limit [1]. For health data, random surrogate assignment held in a separate, access-controlled vault is the safer pattern.

Pseudonymization is not anonymization. Under the GDPR, data that has been pseudonymized but could be attributed to a person with additional information is still treated as information about an identifiable person, and only truly anonymous information falls outside data protection principles [5]. Plan your controls, retention and access as if the corpus still contains personal data, because legally it often does.

What to keep, what to replace, what to generalize

Keep what users search for and what identifies no one. Product names, SKUs, part numbers, error codes, stack-trace symbols, API endpoints, version strings, public companies, regulators and statutes are the anchor terms of most enterprise queries. Replacing them buys little privacy and costs a lot of recall.

Replace direct identifiers: names, emails, phone numbers, street addresses, account and policy numbers, internal ticket IDs that resolve to a customer, IP addresses, device serials tied to a person, and URLs containing user IDs. For health records, the HIPAA Safe Harbor method lists 18 identifier types to remove, including dates (except year) and most geographic subdivisions smaller than a state, and Expert Determination is the alternative when you need to keep more [1]. Treat query logs the same way if you license them with the corpus.

Generalize quasi-identifiers that are harmless alone but identifying together. Job title plus small office plus exact date can single out one employee. Shift dates by a consistent per-entity offset so intervals ("three days after the outage") survive, and bucket rare roles; for health records, date shifting alone does not meet Safe Harbor, which removes all date elements except year, so shifted dates need Expert Determination [1]. NIST's survey of de-identification documents that some de-identified data has been re-identified [4], and its later guidance cautions that traditional techniques have inherent limits compared with formal privacy methods [3].

Entity treatment table

Illustrative example: invented to show structure; it does not describe an available dataset.

Metadata that makes the transform auditable and reversible for deletion

Every chunk should carry the list of pseudonymized entity IDs it contains, the de-identification method version and the mapping version. Vendor guidance on RAG privacy notes that deletion from a vector store depends on tagging at ingestion and querying by metadata [7]. If a data subject or supplier later asks you to remove an entity, you can filter on pseudo_entities and delete or re-embed exactly the affected chunks.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "chunk_id": "tkt-88213#c03",
  "doc_id": "tkt-88213",
  "source_system": "helpdesk",
  "text": "PERSON_0412 at ORG_0077 reported error E-417 on Model X200 ...",
  "pseudo_entities": ["PERSON_0412", "ORG_0077"],
  "deid_method": "typed-surrogate-v3",
  "mapping_version": "2026-09-30",
  "date_shift_applied": true,
  "residual_risk_flags": ["free_text_location_generalized"]
}

Keep the mapping vault outside the delivered corpus and outside your retrieval stack. If the supplier holds the mapping, your license should say so. These fields extend the baseline in metadata a licensed RAG corpus should ship with.

Measuring retrieval loss before and after de-identification

Measure the cost directly: run the same retrieval evaluation on the original and the de-identified corpus and compare. This is a working hypothesis rather than a published standard, but it is the only way to know whether your treatment choices hurt. It usually has to run on the supplier's side or in a clean room, because you should not hold the identified original.

1. Pin both snapshots. Freeze the identified and de-identified corpora with the same chunking and embedding model, as described in pinned corpus snapshots for reproducible RAG evaluation.
2. Rewrite queries through the same mapping. An evaluation query that names "Maria Ortiz" must become "PERSON_0412," or you are measuring query mismatch, not corpus damage.
3. Split the query set. Score single-document lookups, multi-document joins, entity-centric questions and product or error-code questions separately. Damage concentrates in the joins.
4. Compare recall@k, nDCG@10 and MRR, then answer accuracy. Set the tolerance you will accept before you look at the numbers.
5. Inspect the losers. For every query that dropped out of top-k, check whether a keep-list term was wrongly replaced, a surrogate collided, or a quasi-identifier was over-generalized.

Evaluation data needs the same discipline; see de-identifying evaluation data without breaking the test.

Residual risk when the corpus feeds training as well as retrieval

Residual identifiers matter more once the corpus is used for fine-tuning, because what a model memorizes can come back out. Research has shown that training data can be extracted from both open and production language models by querying them [8]. A name that the detector missed in one email signature may be retrievable from a RAG index today and emitted by a fine-tuned model later.

Combining corpora raises a second risk. A pseudonymized support corpus joined with a pseudonymized sales corpus can re-link people through shared quasi-identifiers, even with different surrogate tables. Read linkage and mosaic risk when combining de-identified datasets before you merge, and see retrieval-time privacy controls for personal data in RAG corpora for controls at query time.

A de-identification spec to send with your corpus request

Put the retrieval requirements in writing before the supplier transforms anything, because a corpus redacted the wrong way usually cannot be fixed downstream.

Illustrative example: invented to show structure; it does not describe an available dataset.

• Consistency scope: one mapping across all documents, attachments, metadata fields and delivered evaluation queries.
• Surrogate style: typed tags or realistic surrogates; reserved email domains; format-preserving IDs.
• Keep-list: product names, SKUs, error codes, public organizations, statutes; supplier to confirm additions.
• Replace and generalize lists: direct identifiers, quasi-identifiers, date-shift policy.
• Health data: Safe Harbor or Expert Determination named, with the determination documented.
• Chunk metadata: pseudo_entities, method version, mapping version, residual-risk flags.
• Mapping custody: who holds the vault, who can access it, and how deletion requests propagate.
• Evidence: detector recall on a hand-labeled sample, plus before/after retrieval metrics on a pinned query set.

How SourceX handles de-identification in sourced corpora

SourceX sources operational datasets, such as support and sales histories, engineering records and documents, from US companies on request, and it does not hold them in stock. Every dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery. Personal details such as names, emails, phone numbers and account numbers are removed or replaced before delivery, the method is recorded, and a sample is checked, though no method is perfect. Health records require HIPAA de-identification by Safe Harbor or Expert Determination. You can bring a spec like the one above to the SourceX buyer intake, and see how document corpora are used in RAG evaluation datasets from real company documents.

Request a retrieval-ready de-identified corpus

Describe the documents, the questions your RAG system must answer and your de-identification requirements, and SourceX looks for US businesses that hold that data. Each release is approved by the supplying company, and diligence materials on source, rights, preparation and allowed use are prepared per dataset. Describe the de-identified corpus you need.