De-identifying clinical free text for LLM training: PHI detection, surrogates and residual risk

This page covers what a buyer should accept from a supplier's pipeline. For the preparation side, see the guide to de-identifying medical records; for the broader framework, start at the de-identified data for AI training hub.

Why free-text PHI needs an NLP pipeline, not regex

Free text defeats pattern matching because identifiers appear in irregular forms, inside sentences, with no field boundaries. A structured EHR export has a patient_name column you can drop; an encounter note says "pt's daughter Maria drove her in from Lodi" and hides a relative's name and a small place in prose. HIPAA Safe Harbor covers identifiers of the individual and of relatives, employers and household members, so a pipeline that only targets the patient's own name fails the standard [1].

Mature pipelines are hybrids: pattern matching for well-formed items such as phone numbers and MRNs, machine-learning named-entity recognition for names and places, and a replacement step. Open toolkits such as Microsoft Presidio follow the same recognizer-plus-anonymizer design, and the project itself warns that its trained models cannot guarantee finding all sensitive information [5].

LLM-based redaction is now a serious option. Research on LLM redaction reports that models can catch PII types that evade regex and traditional NER, while also introducing new failure modes that need their own evaluation [3]. For a comparison of approaches, see LLM-based PII redaction vs NER and regex.

What the PHI entity inventory must cover in clinical notes

The detection schema should be broader than the 18 Safe Harbor identifiers, because notes carry identifying detail that the list does not name. Shared-task corpora used to benchmark clinical de-identification have annotated a wider set than HIPAA lists, including professions, full dates, and the names of clinicians and facilities, because combinations of individually harmless details across a longitudinal record can still single out a patient. HHS guidance likewise notes that Safe Harbor fails if the covered entity has actual knowledge that remaining information could identify someone [1].

For a buyer, the practical schema has these layers:

• Direct identifiers: patient, relative and clinician names; MRN, account, health plan and device numbers; phone, fax, email, URL and IP addresses; SSN.
• Dates and ages: all date elements more specific than year, and ages over 89 under Safe Harbor [1].
• Locations: street address, city, ZIP code, facility names, unit names ("4 West"), and named employers.
• Contextual identifiers: rare diagnoses, unusual occupations ("retired state senator"), named events, distinctive family structures, and free-text references to media coverage.

Contextual identifiers are where most residual risk lives. NER models are trained on span-level labels for names and numbers; they rarely flag "the only pediatric lung transplant in the county this year." See indirect identifiers in business text for a fuller taxonomy, and the PHI glossary entry for definitions.

Surrogates vs placeholder tags: how the choice changes your fine-tuned model

Realistic surrogates hide missed PHI among fake PHI, while placeholder tags expose every miss and teach the model a tag vocabulary. The clinical NLP literature calls the surrogate approach "hiding in plain sight": because both manual and automated removal leave residual identifiers, replacing every detected identifier with a realistic fictitious one makes the leftovers hard to tell apart from the surrogates. With tags, a missed "Mrs. Alvarez" stands out in a note where every other name reads [NAME].

The trade-off matters for SFT. Placeholder tags such as [NAME] or <DATE> produce outputs full of tags, which is fine for coding assistance but awkward for summarization or documentation agents that must write natural notes. Surrogates keep fluency, but naive generation can introduce distribution artifacts: gender-mismatched names, impossible date intervals, or one surrogate pool reused so often the model memorizes it.

Consistency is a design decision, not a default. A consistent mapping (same real name always becomes the same surrogate within a patient) preserves coreference across a longitudinal record, but it also means one leaked real name sits next to a stable, repeated pattern that an adversary can exploit; random per-mention substitution breaks coreference instead. Ask the supplier which strategy they used, at what scope (note, patient or corpus), and how date shifting preserves intervals. Keyed mapping options are covered in pseudonymisation techniques for training data.

Acceptance evidence to request from a supplier's pipeline

The evidence that matters is measured recall per entity type on notes like yours, not a vendor's headline F1 on a public benchmark. Public shared-task corpora are useful for system comparison, but notes from a different specialty, EHR template or dictation style can shift performance sharply. Ask for metrics computed on a labeled sample drawn from the delivered corpus itself.

Illustrative example: invented to show structure; it does not describe an available dataset.

For sample sizes and thresholds, use auditing residual PII in a delivered dataset. For the complete document set, use the de-identification evidence package checklist.

Illustrative example: invented to show structure; it does not describe an available dataset.

A hypothetical acceptance note (invented numbers) might read: "On 500 randomly sampled delivered notes, two reviewers found 3 residual direct identifiers (2 relative first names, 1 facility unit phone extension); per-entity recall on the 200-note labeled set was lowest for PROFESSION and LOCATION." That level of specificity lets you decide whether to accept, request a rerun, or add your own scrubbing pass before training.

How residual PHI becomes model risk after fine-tuning

Residual identifiers that survive de-identification can be reproduced by the model you train, which turns a data-handling issue into a product issue. A 2025 study reports a small open model emitting email addresses, phone numbers and account details when prompted, illustrating how personal data in training text resurfaces at inference [4]. Clinical SFT sets are small and repetitive compared with pretraining corpora, which raises the chance that a rare note is memorized.

Three controls reduce exposure. Deduplicate notes and copy-forward templates before training, since repeated text is more likely to be memorized (near-duplicate detection with MinHash and LSH). Run canary or extraction probes against the fine-tuned checkpoint (training-data extraction and memorization risk). And retest the de-identified text with an LLM adversary, since de-identification that resists a human reader may not resist a model (LLM-assisted re-identification).

Safe Harbor, Expert Determination and the legal floor

HIPAA treats health information as de-identified when there is no reasonable basis to believe it can identify an individual, met either by Safe Harbor or by Expert Determination [2]. Safe Harbor also requires that the covered entity have no actual knowledge that remaining information could identify the person, which is hard to certify for narrative text full of context [1]. Expert Determination lets a qualified expert assess risk given the recipient and controls, which often fits training data that needs to keep clinical detail; see Safe Harbor vs Expert Determination for AI training data.

Neither method makes risk zero. NIST SP 800-188 cautions about the inherent limitations of traditional de-identification and stresses governance around releases [6]. State law can add obligations: California Civil Code 1798.148 restricts re-identifying deidentified patient information, so expect license terms that prohibit it (re-identification prohibition clauses) [7].

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Note types and the failure modes to test for each

Each clinical text type fails differently, so the evaluation sample should be stratified by source. The table below lists the patterns to probe.

Illustrative example: invented to show structure; it does not describe an available dataset.

How SourceX handles clinical text requests

SourceX sources operational datasets from US companies on request and manages licensing; it does not hold clinical notes in stock, and a request does not guarantee a match. Health records require HIPAA de-identification by Safe Harbor or Expert Determination, every dataset is rights-reviewed for ownership and consents, and personal details are removed or replaced before delivery with the method recorded and a sample checked. No de-identification method is perfect, which is why the acceptance evidence above still belongs in your review. You can describe the notes you need on the buyer request page or review the medical records licensing overview.

Request de-identified clinical notes for LLM training

SourceX looks for US businesses that hold the clinical or administrative notes you describe, assesses the data and licensing permissions, and agrees allowed uses in a license before anything is delivered through a private, access-controlled workflow. Every release is approved by the supplying company. Describe your note types, volume and de-identification requirements at sourcex.si/buyers.