Automated PII detection vs human review: what each catches

What does automated PII detection do well?#

Automated PII detection does well on identifiers that follow a pattern or appear in a predictable form. Email addresses, phone numbers, payment card numbers, government ID formats and IP addresses can be found with regular expressions and checksums, and named-entity recognition models add names, places and organizations.

Presidio, an open-source SDK for PII identification and anonymization, is a typical example of the approach: it combines named-entity recognition, regular expressions, rule-based logic and checksums with surrounding context. It began as a Microsoft project and is now community-governed under the Data Privacy Stack organization. Cloud services from the major providers work on similar principles.

The strengths are speed, consistency and coverage. A detector applies the same rules to the first record and the last, never tires, and can run again in full after every configuration change.

What human review catches that software misses#

Human review catches identity that lives in meaning rather than format. Reviewers who know the business recognize a customer described by role and location, an employee referred to by nickname, or an incident so specific that anyone in the industry would know who was involved.

Reviewers also catch the opposite problem: words flagged as personal data that are actually product names, error codes, part numbers or place names inside an address the business owns. Leaving those decisions to software alone tends to produce either leaks or hollowed-out records. Give reviewers a short written guideline with examples drawn from your own records, so that two reviewers make the same call on the same ticket.

• 1Indirect references such as the only distributor in a named county or the new plant manager.
• 2Nicknames, initials and internal handles that detectors do not recognize as names.
• 3Customer company names written in lowercase or abbreviated.
• 4Sensitive details about a person without any identifier, such as health or family circumstances.
• 5Identifiers inside screenshots, scanned PDFs and other attachments.
• 6Over-redaction of product names, error messages and part numbers that carry the record's value.

Automated detection vs human review, side by side#

Automated detection and human review differ on almost every operating dimension, which is why they work best together. The comparison below reflects how preparation teams generally experience each.

Why no tool claims complete coverage#

No responsible tool claims complete coverage, because automated detection is probabilistic. Presidio's own documentation says that because it uses automated detection mechanisms, there is no guarantee it will find all sensitive information, and that additional systems and protections should be employed.

That warning applies to every detector. Accuracy varies by record type, language and writing style, so results on clean sample text say little about a decade of support tickets typed in a hurry. The only reliable measure is a review of the output on your own records.

Performance also drifts within one archive. Records exported from a legacy system, messages written in Spanish or other languages, and call transcripts with spoken numbers each behave differently, so measure each slice separately rather than reporting a single overall result.

The working rule: automated first pass, sampled human review#

The working rule is to run automated detection on every record, then have people review a sample from each record type and escalate whole sets to full review when the sample shows risk. The sample is drawn deliberately, not at random from the whole archive, so that every system, period and channel is represented.

• 1Profile record types first: tickets, emails, chat, CRM notes, code comments, attachments.
• 2Configure detectors with custom patterns for your ticket numbers, account IDs and internal codes.
• 3Add an allow list for product names, part numbers and error codes that must not be removed.
• 4Run the automated pass on everything and log what was changed.
• 5Draw review samples from each record type, system and time period.
• 6Fix the detector for every pattern of miss or false positive, then re-run and re-sample.
• 7Send record types with sensitive content or repeated misses to full human review.
• 8Record the sampling plan and results for the buyer's documentation.

Secrets and credentials need their own scan#

Secrets and credentials need their own scan because PII detectors are not built to find API keys, tokens and passwords. Engineering records are full of them: pasted configuration files in tickets, keys committed and later removed from a repository, connection strings in incident notes.

Dedicated secret scanners exist for this. Gitleaks, an MIT-licensed tool, detects passwords, API keys and tokens in git repositories and files, though its maintainer announced in May 2026 that it is feature complete and will receive security patches only; TruffleHog, an AGPL-licensed scanner, can also log in to confirm whether a detected secret is still live. That verification sends real authentication requests, so run it with care and with the system owner's agreement, and rotate any live credential you find.

Illustrative: a software company tunes its first pass#

Illustrative: a fictional B2B software company prepares Intercom conversations and GitHub pull request discussions for a license. Its first automated pass masks emails, phone numbers and most customer names.

Reviewers sampling each year of conversations find customer names inside attached screenshots, a client executive referred to only by first name and title, and a product module named after a common surname that the detector had masked throughout. The team adds the module name to the allow list, routes all attachments to manual handling, and adds a rule for titles paired with first names. A secret scan of the repositories finds old test keys, which are rotated and removed. The second sample comes back clean.

How SourceX combines the two#

SourceX uses both in the Preparation step of the SourceX five-step transaction: Supply, Rights, Preparation, Approval and Delivery. Automated detection runs first, sampled human review follows, and the supplier approves reviewed samples before any release.

The privacy record in the SourceX Evidence Packet describes the tools, the sampling approach and how findings were resolved, so a buyer can see how the records were checked rather than relying on a tool name.