Privacy and preparation
Redaction validation sampling formats: where a random draw goes blind
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A redaction validation sample should be drawn by format, not only by record: census the package by content, then give nested tables, screenshots, scans and unusually encoded text their own quota, search method and pass rule. In a 17-case test of one invented account number, a plain ASCII pattern found 2 variants and a decode-and-normalize search found 15; a split table cell and lookalike letters beat every search.
Redaction
Redaction validation
Key takeaways
- A pooled pass rate can hide a format the sample never touched: a format at 0.5 percent of records is absent from 22 percent of 300-record random draws.
- Build the format census from file signatures and content, because extensions misreport scans, UTF-16 exports and tables stored inside cells.
- Of 17 encodings of one invented number, a plain ASCII pattern found 2, NFKC normalization 6, full normalization 11 and normalization plus decoding 15.
- NFKC maps only 70 of Unicode 15.1.0's 670 non-ASCII decimal digits to ASCII, so map digits by numeric value before searching.
- Split table cells and lookalike letters defeat text search entirely; they need structure-aware rejoining and human viewing.
- A clean format-stratified sample shows redaction reached every format; it does not make data anonymous or licensable.
What should a redaction validation sample prove about difficult formats?
It should prove that every format in the final package was searched by a method able to see it, and that each format passed its own rule. A pooled pass rate cannot show that, because a draw of a few hundred records can contain no screenshot, no scanned page and no file in an unusual encoding, and still report a clean result.
The decision here is narrow: before a redaction validation sample is signed off, which formats need their own quota and their own search method? The evidence that supports a useful answer is a format census of the actual package, a draw from every format the census finds, a search that can read that format, a pass rule written before review and a check that the redacted file still carries what the buyer's task needs.
Four formats create blind spots that a plain-text check cannot see: nested tables, screenshots, scans and text in unusual encodings. Each hides identifiers from a different step. A table can split one value across two cells, a screenshot has no characters to search, a scan may carry an invisible text layer under the image, and an encoding can write an ordinary account number in characters a pattern was never told about.
Passing this test is not the same as being anonymous. The UK regulator's anonymisation guidance says "Identifiability exists on a spectrum," and its published answers to innovators say that removing direct identifiers such as a name or identification number is not enough on its own for effective anonymisation under UK GDPR. A format-stratified sample checks that redaction ran everywhere; whether what remains can identify someone is a separate review, and counsel decides how the law applies to a specific dataset.
How often does a random draw skip a format entirely?
Often, whenever the format is rare. If a format makes up a share p of the records and the sample is a simple random draw of n records from a much larger package, the chance that the draw contains none of that format is (1 - p) raised to the power n. The table works it out for four shares and two sample sizes, rounded half up to a whole percent, or to one decimal place below 1 percent.
A format at half a percent of the package is missing from 61 percent of 100-record draws and 22 percent of 300-record draws. Even when the format does appear, one or two items cannot support a claim about it. The number of clean items needed for a given confidence is a separate calculation, covered by the rule-of-three sample sizes in the live guide linked below; this page is about making sure each format gets items at all.
The fix is to treat format as a stratum, with a quota of its own, rather than hoping the random draw finds it. That changes the reporting too: each stratum gets its own count of items checked and identifiers found, and a stratum the census found but the sample never touched is reported as unchecked, not as clean. The table below gives the arithmetic:
| Format share of records | Chance a 100-record draw has none | Chance a 300-record draw has none | Expected items in 300 |
|---|---|---|---|
| 0.2 percent | 82 percent | 55 percent | 0.6 |
| 0.5 percent | 61 percent | 22 percent | 1.5 |
| 1 percent | 37 percent | 5 percent | 3 |
| 2 percent | 13 percent | 0.2 percent | 6 |
Find the formats from what the bytes say, not the file name
Build the census from file signatures and content, because extensions and system labels misreport exactly the formats that matter. A .pdf can be a born-digital document with a clean text layer or a stack of scanned images; a .txt export can be UTF-16; a .csv can hold a JSON object or an HTML fragment in one cell; an email body can arrive quoted-printable or base64 encoded.
For each file, record the detected type from its leading bytes, whether a text layer exists and how much text it holds per page, the character encoding and any byte-order mark, the maximum table nesting depth, and whether the text contains characters outside the scripts the business normally writes in. Those few fields are enough to assign every file to a stratum before anyone reads a record.
Count records per stratum and keep the counts with the sample plan. They are what lets a reviewer say later that a stratum held, for example, 412 records of which 50 were checked, or that it held none. Without them, a missing format and a clean format look the same in the report. The signals that place a file in each difficult stratum:
- Nested tables: HTML or XML tables inside table cells, spreadsheet ranges with merged or stacked headers, delimited files whose cells hold JSON, HTML or line breaks.
- Screenshots and photos of screens: image files, and images embedded in tickets, emails and documents, found by file signature rather than by extension; a reviewer tags which ones are screen captures.
- Scans and image-only PDFs: pages whose text layer is empty or far shorter than the rendered page suggests, and pages with an invisible OCR layer under the image.
- Unusual encodings: files that are not UTF-8, byte-order marks, quoted-printable or base64 parts, HTML character references, digits outside 0 to 9, no-break or narrow spaces and zero-width characters inside numbers.
Which encodings slip past the validation search? A 17-case test
Most do, unless the search decodes and normalizes first. I wrote one invented account number, 4821 7730 0915, in 17 ways that occur in business exports and ran five searches over each. A plain ASCII pattern found 2 of the 17. The strongest search, which decodes transfer encodings and normalizes characters before matching, found 15. Two variants beat all five: the number split across two table cells and the number typed with lookalike letters.
The five searches build on each other, and their totals were 2, 8, 6, 11 and 15 of 17 for P1 to P5. P1 is a pattern of digits 0 to 9 with an optional space or hyphen between groups, run on the raw text. P2 swaps in Python's Unicode-aware classes, where, as Python's documentation explains, \d matches any character in the Unicode Nd category in a str pattern. P3 applies NFKC normalization, the compatibility decomposition defined in Unicode Standard Annex 15, and then runs P1. P4 also drops default-ignorable characters, maps every decimal digit to its ASCII value and turns Unicode spaces and dashes into ASCII ones. P5 also decodes HTML character references, quoted-printable soft line breaks, base64 runs and UTF-16 bytes before running P4.
Each step closes a different gap. P2 found Arabic-Indic and Devanagari digits that NFKC leaves alone, while NFKC caught no-break spaces but missed a non-breaking hyphen, which it maps to a Unicode hyphen rather than an ASCII one. Neither found a zero-width space or soft hyphen inside a group; only P4's removal of default-ignorable characters did. A quoted-printable soft line break, which RFC 2045 uses to keep encoded lines within 76 characters, split the number in the raw email source until P5 decoded it.
The character data explains why this is not a curiosity. In the Unicode 15.1.0 Character Database, 680 code points are decimal digits and 670 of them are not ASCII. NFKC maps 70 of those 670 to ASCII digits, the fullwidth, segmented and five mathematical styles, and leaves the other 600 unchanged. Another 30 characters, superscripts, subscripts and circled digits, also normalize to a single ASCII digit. Unicode lists 25 white-space characters, of which a pattern that allows only the ASCII space accepts one, and 405 assigned characters with the Default_Ignorable_Code_Point property, the class that holds the zero-width space and soft hyphen used in the test.
The two survivors need different tools. The split-cell variant needs a structure-aware check that rejoins adjacent cells, which is why nested tables get their own stratum. The lookalike variant needs a visual comparison: under the skeleton procedure in Unicode Technical Standard 39, 210 other assigned characters reduce to the same skeleton as an ASCII digit, including the Latin capital O for zero and the lowercase l for one, and a person reading the record sees a number where a pattern sees letters. The table groups the 17 variants into six families:
| Variant family | Variants | P1 ASCII raw | P4 normalized | P5 normalized and decoded |
|---|---|---|---|---|
| ASCII digits with spaces or hyphens | 2 | 2 of 2 | 2 of 2 | 2 of 2 |
| Unicode spaces and hyphens | 3 | 0 of 3 | 3 of 3 | 3 of 3 |
| Other digit forms: fullwidth, mathematical, Arabic-Indic, Devanagari | 4 | 0 of 4 | 4 of 4 | 4 of 4 |
| Invisible characters: zero-width space, soft hyphen | 2 | 0 of 2 | 2 of 2 | 2 of 2 |
| Transfer encodings: HTML references, quoted-printable, base64, UTF-16 | 4 | 0 of 4 | 0 of 4 | 4 of 4 |
| Structure and lookalikes: split cells, letters for digits | 2 | 0 of 2 | 0 of 2 | 0 of 2 |
The format-stratified validation matrix
The matrix gives each difficult format a stratum, a way to find it, a draw and search method, a pass rule and the utility check that keeps redaction from destroying what a buyer needs. The floors are editable defaults: 50 items per stratum, or every item when a stratum holds fewer. A clean draw of 50 rules out a stratum-wide failure but supports only a loose bound, about 6 percent by the rule of three; a claim near 1 percent needs the 300 clean items the live sampling guide describes.
Run the plain-text stratum as a control. If misses cluster there, the detector is weak; if they cluster in a difficult stratum, the method cannot see that format. The two problems have different fixes, and a pooled number hides which one you have. Here is the matrix:
| Format stratum | How the census finds it | Draw and search method | Pass rule | What must survive redaction |
|---|---|---|---|---|
| Plain UTF-8 text (control) | Everything not placed in another stratum | Random draw per the record-type plan; normalized and decoded search | No identifier in the redacted text | Wording, codes and outcomes the task reads |
| Nested tables | Table depth above one; cells holding JSON, HTML or line breaks; merged headers | Editable floor of 50; search each cell, then rejoin adjacent cells and search again | No identifier in any cell or across a cell boundary; header labels naming people count | Row and column structure, header-to-value links and totals |
| Screenshots and photos of screens | Image signature in attachments and embedded images; reviewer tags screen captures | Editable floor of 50; a person views each at full size and the OCR text is searched | No readable identifier in pixels, including tabs, address bars, banners and cropped text | The screen state the task needs, such as an error message or a status |
| Scans and image-only PDFs | Empty or thin text layer per page; invisible OCR layer under an image | Editable floor of 50 pages; render and view, search the OCR output and any hidden layer separately | Rendered page and every text layer clean; a box drawn over text still in the layer fails | Page order, layout, stamps and annotations that are not personal |
| Unusual encodings | Not UTF-8, byte-order mark, quoted-printable or base64 parts, entities, non-ASCII digits or invisible characters | Editable floor of 50; decode, normalize, map digits by value, then search; keep a raw-byte pass | Clean after decoding and normalizing; a find that appears only after decoding is a miss | Non-personal text in other scripts, such as product names |
Is format coverage the real gap, or is the detector just weak?
Sometimes the detector is the problem, so test for it rather than assume formats explain every miss. The plain-text control stratum separates the two: a detector that misses names in ordinary sentences will miss them in every format, and adding strata will only spread the misses around. When the control passes and a difficult stratum fails, the method is blind to that format, and tuning the detector will not help.
The stronger objection is that format coverage solves the wrong problem. A package can pass every stratum and still identify people through combinations of job titles, dates, places and rare events, and it can be perfectly redacted and still carry content that a contract restricts. Both are true. Format coverage answers one question, whether redaction ran on everything; indirect identification and licensing rights need their own reviews, and neither is settled by a clean sample.
So read the result in three parts. Strata that passed tell you the method reached every format. A control that passed tells you the detector works on ordinary text. Neither tells you the data is anonymous or licensable, and the review record should say so plainly.
Final-delivery checklist for format coverage
Hold the delivery until every item below is true for the package that will actually ship. Each item is something a reviewer can check from the sample record without rereading the data:
Checklist · 8 items
- A format census of the final package exists, built from file signatures and content, with a record count for every stratum.
- Every stratum the census found was drawn; a stratum with no items is recorded as zero, and a stratum the draw missed is recorded as unchecked.
- The validation search decoded transfer encodings and normalized characters before matching, and its version and settings are logged.
- A person viewed every sampled screenshot and scanned page at full size, and their OCR text went through the same search as the body text.
- Table values were searched cell by cell and again with adjacent cells rejoined.
- Results are reported per stratum, with items checked and identifiers found, never as one pooled rate.
- Any miss reopened its whole stratum, the fix was applied to the stratum, and a fresh draw passed.
- The utility fields listed for each stratum were spot-checked in the redacted files, and the sampled files match the delivered files by hash.
0 of 8 ticked. Tick each item as you go.
Methods note: data, versions and what the test leaves out
Character counts come from four Unicode 15.1.0 data files copied from the unicode-org/unicodetools repository at commit 578f6c0 (UnicodeData.txt, DerivedCoreProperties.txt, PropList.txt and confusables.txt), read on 2026-10-28. Decimal digits are code points with General_Category Nd. NFKC and NFD come from Python 3.13's unicodedata module, which reports Unicode 15.1.0. A code point counts as NFKC-mapped when its NFKC form is exactly one of the characters 0 to 9. The default-ignorable count includes only code points that UnicodeData.txt assigns. The skeleton follows the steps in Unicode Technical Standard 39: NFD, remove default-ignorable characters, replace each character with its prototype from confusables.txt, NFD again; a code point other than the ASCII digits counts when its skeleton equals the skeleton of an ASCII digit.
The scored set uses one invented number, 4821 7730 0915, inside one invented sentence, written 17 ways. A search finds a variant when its pattern matches a span whose digits, read by decimal value, equal 482177300915. P1 uses the pattern [0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4} with ASCII-only matching and no digit directly before or after; P2 uses \d and \s in place of the digit and space classes. P3 and P4 apply NFKC first; P4 then removes every code point with the Default_Ignorable_Code_Point property, maps each General_Category Nd character to its decimal value, each White_Space character to an ASCII space and each dash-punctuation (Pd) character to an ASCII hyphen. P5 tries the raw text, the text with HTML references decoded, the quoted-printable decoding (characters outside Latin-1 are dropped before that step), every base64 run of 16 or more characters that decodes to UTF-8 and, when the text holds a null character, a UTF-16LE reading. The coverage table uses (1 - p) to the power n, which assumes a package much larger than the sample.
Not measured: the share of any real package in each stratum, OCR accuracy on real screenshots or scans, any commercial or open-source detector, and how often lookalike or split values occur in business records. The 17 variants show which search steps matter; they are not a sample of real data. The scripts and their output are kept for recount, and no personal data was used.
Preparing records for a SourceX license
If you are considering a license, the format census is useful before any agreement, because it shows how much of your archive is plain text and how much is tables, images, scans or odd encodings. That shapes scope and preparation effort, and you can run it without showing anyone a record.
SourceX describes its own handling on its trust page in these words: once SourceX gives the go-ahead, the data is "extracted, transferred and de-identified through approved processing routes." Its legal framework page adds that "The method used is recorded for each dataset," and that no method is perfect, so de-identification reduces risk without removing it. You decide which systems, records and date ranges are in scope, and the first step is a short company assessment that needs no data upload; it is not an offer.
Frequently asked questions
Does a bigger random sample solve format coverage?
Only slowly and expensively. A format at half a percent of records is still missing from 22 percent of 300-record random draws. A stratum with its own quota makes sure the format is checked, at a fraction of that review effort.
Should screenshots be redacted or excluded?
Decide by what the task needs. If the screenshot carries the evidence a buyer pays for, such as an error state, redact the pixels and check them by eye. If it adds little, excluding the stratum is cheaper and safer, and the census shows how much you give up.
Is NFKC normalization enough before searching for identifiers?
No. In this test NFKC caught fullwidth digits and no-break spaces but left Arabic-Indic and Devanagari digits, a non-breaking hyphen and zero-width characters untouched. Map digits by numeric value, drop default-ignorable characters and decode transfer encodings as well.
How do I find nested tables without reading every file?
Parse the files and count table depth, merged header cells and cells that contain JSON, HTML or line breaks. Those counts come from structure alone, so the census can run before anyone looks at a record's content.
If every stratum passes, is the dataset anonymous?
No. A passed sample shows redaction reached every format. Under UK GDPR the ICO treats identifiability as a spectrum and says removing direct identifiers is not enough on its own, so indirect identifiers need a separate review, and counsel decides how the law applies.
Sources
- Unicode 15.1.0 data files (UnicodeData.txt, DerivedCoreProperties.txt, PropList.txt, confusables.txt) behind the character counts: 680 decimal digits, 670 non-ASCII, 70 mapped to ASCII by NFKC, 25 white-space characters, 405 assigned default-ignorable characters, 210 digit lookalikes by skeleton. Source
- Unicode Technical Standard 39 defines the skeleton used to compare confusable strings: NFD, remove default-ignorable characters, replace characters with prototypes, NFD again. Source
- Unicode Standard Annex 15 defines NFKC as compatibility decomposition followed by canonical composition, which replaces compatibility characters. Source
- Python's re documentation: in str patterns \d matches any Unicode decimal digit (category Nd), and the ASCII flag restricts it to [0-9]. Source
- RFC 2045 limits quoted-printable encoded lines to 76 characters and marks soft line breaks with a trailing equal sign that decoders remove. Source
- ICO anonymisation guidance (UK GDPR): identifiability is a wide concept and exists on a spectrum. Source
- ICO innovation advice (UK GDPR): removing direct identifiers such as a name or identification number is insufficient to ensure effective anonymisation. Source
- SourceX trust page: once SourceX gives the go-ahead, the data is extracted, transferred and de-identified through approved processing routes; the supplier decides scope. Source
- SourceX legal framework page: the de-identification method used is recorded for each dataset; no method is perfect. Source
See if your company qualifies
A short company assessment. No data uploads are needed.