Privacy and preparation
Free text redaction false negatives: testing detectors on signature blocks and footers
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Test identity detectors against signatures and footers with a stratified false negative audit: split each record into zones (body, salutation, signature block, legal footer, quoted reply, system footer), draw a fixed sample from every zone, have two people label every identifier, and count what the redacted copy still exposes per zone. A pooled recall figure hides signature-block misses; a per-zone miss count and upper bound does not.
Redaction
Free text redaction false
Key takeaways
- Pooled recall is dominated by body text and can hide a detector that misses most signature blocks.
- Split records into zones such as signature, footer, quoted reply and system template, and sample each zone on its own.
- Report misses and an upper bound per zone; zero misses in 30 items proves far less than zero in 150.
- Remove low-value zones structurally before detection, and re-audit after every rule or threshold change.
- Names are not the only risk: titles, office locations and profile handles can still single out a person.
How do you test a name detector against signature blocks and footers?
Treat the signature block and the footer as separate test populations, not as a few extra lines of body text. Sample them on purpose, label every identifier a person could read in them, run the detector, and report misses for each zone on its own line.
The reason is arithmetic. In a typical support email or ticket comment, the body holds most of the words, so a recall figure computed over whole records is dominated by body text. A detector can score well overall while leaving a direct dial, a mobile number and a full name in nearly every signature, because those lines are a small share of the tokens and a large share of the exposure.
Recall is the share of real identifiers the tool found: true positives divided by true positives plus false negatives, as the Presidio project's evaluation guide defines it. That guide also notes that recall usually matters more than precision in PII detection and suggests weighting it more heavily. A false negative audit is simply the disciplined way to count the denominator you cannot see from the redacted output alone.
Why does one overall recall number hide the signature-block miss?
A pooled score averages over zones that behave very differently. Body text is conversational, so named-entity models trained on prose do reasonably there. Signature blocks are terse, line-broken and full of fragments such as a first name alone, a title, a company, a street line and two phone numbers with labels like M or D. Footers are boilerplate written in legal register. Quoted replies repeat another person's words and signature, often with a forwarded header block that names them again.
Each of these zones fails for its own reason, and the failures do not cancel out. If a signature block appears in most outbound messages, one missed pattern is repeated thousands of times across an archive. A pooled figure treats that as a small error rate; a reader of the released dataset experiences it as the same named employee appearing on every page.
Tool documentation is candid about this. The Presidio FAQ says automated detection cannot be relied on to find all sensitive information and that every identification approach trades false positives against false negatives. Amazon's service card for Comprehend PII detection describes a fixed set of supported entity types and tells customers to test on their own content. Neither source promises zone-level performance, which is why you measure it.
Which zones of a business record need their own stratum?
Define strata by where text sits in the record and how it was produced, then cross them with the source system. A zone is worth its own stratum when its layout, vocabulary or authorship differs from the body enough that a detector could behave differently there.
Signature blocks deserve the most care because there is no reliable marker for them. RFC 3676 describes the long-standing convention of a dash-dash-space line separating a signature from the message, but many mail clients and most people do not use it, and ticketing systems often strip or merge it. Expect to find signatures by position and pattern, not by a delimiter.
- Body: the text the author wrote in this message or comment.
- Salutation and sign-off: greeting lines and the closing name, often a first name alone.
- Signature block: name, title, company, direct dial, mobile, address, social profile links, pronouns, certifications after a name.
- Legal and confidentiality footer: registered office address, company number, named compliance contacts, disclaimers.
- Quoted reply and forward headers: earlier messages, On date, Name wrote lines, From, To and Cc blocks.
- System footers and headers: ticket footers, sent-from lines, unsubscribe blocks, calendar invite details, auto-replies naming a delegate.
- Document furniture in converted files: letterhead, page headers and footers, fax cover sheets and form labels produced by OCR or PDF extraction.
The stratified false negative audit for business text
Use this table as the audit plan. Each row is one stratum; the sample size is per source system and per audit round, and the release rule is the condition the stratum must meet before the redacted copy moves forward. Counts are illustrative starting points to adjust with your privacy reviewer, not a statistical standard.
| Stratum | How to locate it | What labelers mark | Sample per round (illustrative) | Release rule |
|---|---|---|---|---|
| Body text | Text before the first sign-off, quoted header or footer pattern | Names, contact details, account and order numbers, addresses, rare job or event details | 100 records | No direct identifier missed; indirect details reviewed |
| Salutation and sign-off | First and last two non-empty lines of each message | First names alone, nicknames, initials used as names | 100 records | No name left that maps to a known person in the roster |
| Signature block | Lines after the sign-off, dash-dash-space marker or contact-label pattern | Full name, title, company, direct dial, mobile, address, profile URL | 150 blocks | Zero missed phone numbers or full names |
| Legal and confidentiality footer | Repeated trailing paragraphs found by frequency across the archive | Named contacts, registered address, personal email of a compliance officer | 50 distinct footer variants | Every variant handled by a rule, not by chance |
| Quoted reply and forward header | Lines beginning with quote markers, wrote lines, From and Sent headers | Third-party names and emails, earlier signatures inside the quote | 100 threads | Quoted zone removed or redacted to the body standard |
| System footers and auto-replies | Template strings per system: ticket footers, out-of-office text, invite blocks | Delegate names, phone numbers, meeting links with names | 50 template variants | Templates matched and removed structurally |
| Converted document furniture | Page header and footer bands, letterhead area, form labels after OCR | Names and numbers split across lines or merged with labels | 50 pages per document type | No miss on headers repeated across pages |
How many items per stratum, and what does a clean sample actually prove?
Sample enough that a clean result means something, and say what it means. If labelers find zero misses in n independent items from a stratum, a common rule of thumb puts the approximate 95 percent upper bound on the true miss rate at about three divided by n. With 150 signature blocks and no misses, that bound is roughly 2 percent; with 30 blocks it is roughly 10 percent, which is too loose to support a release.
Draw the sample after stratifying, and draw it at random within each stratum, so that the busiest sender or the most common template does not crowd out the rest. Sample distinct footer and template variants rather than raw occurrences, because one fixed template either is handled everywhere or fails everywhere.
Use two labelers on the same items and reconcile disagreements before scoring. A miss that only one labeler saw is still a miss; a span that both labelers argue about is a definition problem to settle in the labeling guide. Record the labeling guide version with the results so the next round is comparable.
Report per stratum: items sampled, identifiers labeled, identifiers missed by type, the upper bound when misses are zero, and the detector version and settings used. Never merge strata into one figure for the release decision.
Which identifying combinations survive even when every name is caught?
Removing every name and number does not finish the job, because a signature block is a bundle of quasi-identifiers. NIST defines a quasi-identifier as a variable that can identify a person through association with other variables. A title, a small office location and a niche certification can point to one person even with the name gone.
The UK ICO frames this as the motivated intruder test: whether a reasonably competent person with ordinary resources such as the internet and public records could identify someone from the data. Signature residue is exactly what such a person would search: a job title plus a company plus a city often returns a single professional profile.
Label these combinations as their own finding type in the audit. Common survivors are a unique job title at a small company, a branch or office address, a profile URL with the name removed but the handle intact, a personal pronoun line next to a role, a certification string, and a footer naming the only compliance officer. Decide per combination whether to drop, generalize to a role token, or keep with a documented reason.
Worked example: auditing an exported support mailbox (illustrative)
Illustrative only. A field service software company exports two years of support email and ticket comments and runs an NER-based detector configured for names, emails, phone numbers and addresses. A pooled check on 200 random messages finds 94 percent recall, and the team is ready to proceed.
The stratified audit tells a different story. In 150 sampled signature blocks, labelers find 41 blocks where a mobile or direct-dial number survived because it followed a single-letter label, and 12 where a first name on its own line was missed. In 50 distinct footer variants, three name a compliance contact. In 100 quoted threads, 18 still carry a customer's full name in an On date, Name wrote line. Body text holds up: two misses in 100 records, both rare surnames.
The fixes are structural first. The team strips quoted history before detection, removes known footer variants by exact match, and segments signature blocks by position and contact-label patterns, replacing them with a role token such as support engineer. A second round on fresh samples finds zero misses in 150 signature blocks, which supports a stated upper bound near 2 percent for that stratum, and the residual combinations go to the privacy reviewer as a short list.
Should you tune the detector or remove the zone structurally?
Remove a zone structurally when its business value is low and its pattern is stable; tune the detector when the zone carries signal you need to keep. Signatures, legal footers, system templates and quoted history rarely carry task value, so deleting or tokenizing them is usually safer than teaching a model to find every fragment inside them.
When you must keep a zone, use the customization the tool offers and then re-audit. Presidio supports regex and deny-list recognizers with context support for custom entities. Google Sensitive Data Protection lets a hotword rule raise or lower the likelihood of a finding based on nearby words, for example a name near a label. Amazon's service card says Comprehend PII detection works from a fixed entity taxonomy without custom PII types and that the confidence threshold can be tuned toward fewer false negatives or fewer false positives.
Every change goes back through the same strata with a fresh sample. A rule that fixes signatures can create new misses elsewhere, for example by deleting a body paragraph that happened to begin with a name and a phone number.
- Remove: quoted history, known footer variants, out-of-office and system templates.
- Tokenize: sender and recipient names replaced by stable role or person tokens so who-said-what survives.
- Tune and keep: body text where entity mentions matter to the task, with context words and thresholds documented.
- Escalate: zones with free-form personal content, such as notes about health, family or disputes.
Audit results that should pause the dataset or go to a privacy specialist
Pause when any stratum misses a direct identifier after the structural fixes, when a stratum is too small to sample to the planned size, or when labelers cannot agree what counts as identifying in a zone. Each of these means the redacted copy's risk is unknown, not low.
Bring in a privacy specialist when residual combinations remain that you want to keep for utility, when records involve children, health, finances or employment disputes, or when the archive spans jurisdictions with different definitions of personal data. This article is general information; counsel and a privacy reviewer decide on the specific facts.
Keep the audit record with the dataset: strata, sample sizes, labeling guide, misses by type, fixes, second-round results and the remaining accepted risks. That record is what lets a later reviewer check the decision instead of trusting it.
What a licensing review takes from the audit record
A data licensing review asks whether redaction was measured where it tends to fail, so a per-zone audit record shortens that conversation. It also keeps separate questions separate: a clean audit shows privacy preparation, not that the company owns the records or has the contractual right to license them.
SourceX's part in licensing selected business records directly from a company covers rights review, contracting, preparation and delivery. Its de-identification is done through approved processing routes, and redacted text goes nowhere before the company has approved scope and terms and signed. A first conversation starts with a short company assessment that needs no data upload.
Frequently asked questions
What is a false negative in free text redaction?
A false negative is a real identifier that the redaction tool did not find, so it remains readable in the cleaned copy. You cannot see false negatives in the tool's output; you find them by having people label a sample independently and comparing their labels with what the tool caught.
Why do PII tools miss names in email signatures?
Signature blocks are short fragments without sentence context, often with a first name alone, labels such as M or D before numbers, and line breaks that split an address. Models trained mostly on prose see fewer cues there, and the lack of a consistent signature delimiter makes the zone hard to isolate.
How large should a false negative audit sample be?
Size each stratum separately. As a rough guide, zero misses in n items gives an approximate 95 percent upper bound of three divided by n on the miss rate, so 150 clean signature blocks support a bound near 2 percent while 30 support only about 10 percent.
Is it better to delete signature blocks or redact them?
Delete or tokenize them when they carry no task value, which is the usual case, because structural removal does not depend on a detector catching every fragment. Keep and redact only when the role or organization in the signature matters to the dataset, and replace names with stable role tokens.
Does a clean redaction audit make records safe to license?
No. The audit measures how well identifiers were removed in the zones you sampled. Whether the company may license the records depends on ownership, contracts and privacy permissions, which counsel reviews separately.
Sources
- The Presidio FAQ says automated detection cannot ensure all sensitive information is found and that PII identification trades false positives against false negatives. Source
- Presidio's evaluation guide defines recall as TP / (TP + FN) and notes recall is often more important than precision in PII detection. Source
- Presidio's PatternRecognizer supports regex and deny-list recognition with validation and context support for custom entities. Source
- RFC 3676 section 4.3 describes the Usenet and email convention of a dash-dash-space line separating a signature from the message body. Source
- Google Sensitive Data Protection hotword rules adjust the likelihood of a finding depending on whether a hotword occurs near it. Source
- Amazon's AI service card for Comprehend PII detection describes a fixed entity taxonomy without custom PII types, a tunable confidence threshold, and advises testing on your own content. Source
- The UK ICO describes the motivated intruder test, assuming a reasonably competent intruder with ordinary resources such as the internet and public records. Source
- NIST's glossary defines a quasi-identifier as a variable that can identify an individual through association with other variables. Source
See if your company qualifies
A short company assessment. No data uploads are needed.