Redaction QA sampling: how many records should you review?

What does a redaction QA sample measure?#

A redaction QA sample measures how often personal details survive automated redaction, by having people read a random selection of processed records and count the ones with a missed identifier. Nobody can read a full archive, so the sample is how a COO or privacy lead turns a sense of how good the redaction is into a number that can be written down.

Pick the unit before you start. The simplest unit is the record: a ticket, an email thread or a transcript counts as a miss if it contains at least one identifier that should have been removed. Counting individual identifiers is more precise but slower, and record-level counting is what most buyers and counsel ask about.

The sample must be random within each record type. Records chosen by convenience, such as the first page of an export or the shortest tickets, measure nothing useful.

The rule of three: sample sizes at a glance#

The rule of three says that if n randomly sampled records contain zero misses, you can state with 95 in 100 confidence that the true miss rate is below about 3 divided by n. It comes from asking how high the miss rate could be while zero misses in n records would still not be surprising. In acceptance-sampling terms it is a zero-defect plan: a single miss fails the round.

The table turns that into review plans. Choose the claim you need to make, then read across to the number of clean records required.

Why dataset size barely changes the sample#

Dataset size barely changes the sample because the bound depends on the number of records reviewed, not on the share of the dataset they represent. A clean sample of 300 supports the same claim whether the record type is a modest archive or a very large one.

Small record types are the exception. When a record type holds only a few hundred records, the sample becomes a large share of the whole, and reviewing every record is often simpler than sampling. Do that for small but sensitive sets such as HR-adjacent email or executive threads.

Higher confidence costs more records. At 99 in 100 confidence, the rule becomes roughly 4.6 divided by n, so the same claim needs about half again as many clean records.

Dataset size does change what the bound means in absolute terms. In a hypothetical record type of 200,000 tickets, a bound of 1 in 100 still allows for up to about 2,000 tickets with a surviving identifier. That is why counsel may ask for a tighter claim on large free-text record types, or for scope changes that drop the hardest fields.

Stratify by record type and risk#

Stratifying means drawing a separate random sample for each record type, because a single pooled sample lets an easy record type hide a hard one. Redaction that works well on structured CRM notes can fail on call transcripts, and a pooled sample dominated by notes will not show it.

Keep targeted review separate from random sampling. Searching for known hard patterns, such as signatures or spelled-out emails, is a good way to find problems, but those records do not count toward the random sample that measures the miss rate.

• 1Record family: tickets, email threads, chat, call transcripts, CRM notes, code comments.
• 2Source system and era: records from before and after a system migration often format names differently.
• 3Content type: body text, attachments converted to text, and metadata fields.
• 4Risk: oversample strata that are likely to mention health, finances, minors or home addresses.

What counts as a miss?#

A miss is any identifier that the redaction rules say should be removed but that survives in the processed record. Write the rules down before review starts so reviewers do not decide case by case.

What to do when the sample finds a miss#

When the sample finds a miss, the measurement is over for that round, and the job becomes finding the cause. Patching the one record and continuing to count would make the final claim meaningless.

One miss also weakens what the sample can support. With one miss in n records, the upper bound rises to roughly 4.7 divided by n; with two misses, roughly 6.3 divided by n. Fixing the cause and drawing a fresh full sample is usually faster than reviewing more records to recover the claim.

• 1Stop counting and record the miss with its record type and pattern.
• 2Identify the cause: missing pattern, text extraction error, unusual format or reviewer disagreement.
• 3Fix the configuration or add a preprocessing step.
• 4Re-run redaction across the entire record type, not just the sampled record.
• 5Search the whole record type for the same pattern.
• 6Draw a new random sample of full size and start the count again.

Illustrative: a restoration contractor plans its QA#

Illustrative: a fictional water and fire restoration company is preparing job notes, adjuster email threads, estimate line items and photo captions from its job management system. Homeowner names, addresses and insurance claim numbers appear throughout.

The COO sets four strata, plans 300 clean records for estimates and photo captions, and 1,000 for job notes and adjuster emails because those carry the most free text. Two reviewers read an overlapping portion to check that they agree, and the team seeds a handful of records with known identifiers to confirm reviewers catch them.

The first email sample turns up a homeowner's name inside a forwarded signature block. The team adds signature stripping, re-runs the email stratum, and draws a fresh sample that comes back clean. The QA log records both rounds.

How SourceX uses QA sampling#

SourceX builds QA sampling into the Preparation step of the SourceX five-step transaction. Sample sizes per record type, the definition of a miss, each round's results and any fixes are written into the privacy record of the SourceX Evidence Packet, and the supplier reviews that record before giving approval. The claim that goes into the license is the one the samples support, no stronger.