Skip to content

Privacy and preparation

Multimodal privacy audit workflow: finding what text redaction missed

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

To find personal information left in screenshots after text redaction, audit each layer of the record separately and in order: file inventory, record text, text embedded in files, image pixels with OCR and human viewing, audio against the redacted transcript, then metadata and the file container, and finally the record as a whole. Text redaction reads characters only, so a clean text sample says nothing about the other layers.

Redaction

Multimodal privacy audit

This page covers multimodal privacy audit workflow: finding what text redaction missed.Illustrative

Key takeaways

  • Text redaction reads characters, so it cannot see names in screenshots, scans, audio or metadata.
  • Audit each layer in order: inventory, text, embedded text, pixels, audio, metadata and container, then the record as a whole.
  • Run the metadata and container check last, on the final files, because redaction tools rewrite files and PDFs can keep old revisions.
  • Automated detectors are incomplete, so every stage ends with a human check and a written exit criterion.
  • Excluding an attachment type is often safer and cheaper than redacting it.

How do you find personal information left in screenshots after text redaction?

You find it by auditing every layer of a record separately, in a fixed order, because text redaction reads characters and never sees pixels, sound or file metadata. A multimodal privacy audit workflow checks the text, then the text hidden inside files, then what images show, then what recordings say, then the metadata and container, and finally what the layers reveal in combination.

Each layer needs its own detector and its own human check. Google's Sensitive Data Protection documentation, for example, describes using optical character recognition to find text inside an image and then analyzing that text as it would any other text. A pipeline that skips the OCR step will pass a screenshot showing a customer's full name in a CRM header as clean, because to the text redactor the screenshot is just a file reference.

The goal of the audit is a record that says, layer by layer, what was checked, how, by whom and what was found. That record is what lets a privacy reviewer approve the dataset rather than hope the text pass was enough.

Why does clean text not mean a clean record?

Clean text does not mean a clean record because a support ticket, email or job record is a bundle of layers, and each layer is stored and processed differently. The body text sits in a database field. A screenshot sits in file storage as pixels. A PDF quote has a visible page, a text layer, annotations and an internal revision history. A voicemail is audio. Every one of those files carries metadata that no reader sees on screen.

Tools reflect that split. Microsoft Presidio's image redactor detects and redacts PII text in images using Tesseract OCR, and its documentation notes that its DICOM engine redacts only pixel data and does not scrub text that may sit in the image metadata. Pixel redaction and metadata cleaning are separate jobs, and an audit that checks one has not checked the other.

NIST's report on de-identifying personal information treats non-tabular data, such as free-format text and images, as its own challenge for this reason. The identifying details in a photograph or a scanned page do not live in labeled fields, so field-based rules cannot reach them.

The text, image, audio and metadata review sequence

The sequence below is the reusable asset. Run it on a processed sample of each record type, in order, and do not start a stage until the previous one has met its exit criterion. Stages that do not apply to a record type, such as audio for a ticket queue with no recordings, are marked not applicable in the record rather than skipped silently.

Sample within each stage by record type and by file type, not across the whole archive. A random sample of tickets may contain few voicemails or scanned forms, so draw the stage 3 and stage 4 samples from the attachments themselves. Stage 3 is the one place where the sample should be viewed in full: every image in the sample is opened at full size, not skimmed as thumbnails.

Write the miss rules before review starts. Decide, for example, whether a first name alone counts, whether a company logo in a screenshot counts, and whether a staff member's name in a browser tab counts. Reviewers who decide case by case produce results that cannot be compared between rounds or between people.

The text, image, audio and metadata review sequence
StageLayerWhat to look forHow to checkExit criterion
0 InventoryEvery file type attached to the record typeScreenshots, photos, scans, PDFs, spreadsheets, documents, audio, video, archives, logsCount attachments by type and size from the export, before any reviewEvery type has a decision: exclude, redact and audit, or keep as is
1 TextRecord fields, bodies, notes, subject linesNames, emails, phones, addresses, account and order numbers, free-text mentionsAutomated detection followed by a human read of a random sampleNo missed identifiers in the sample under your written miss rules
2 Embedded textText inside files: PDF text layers, form fields, annotations, document comments, spreadsheet cells and hidden sheetsIdentifiers that a viewer does not show, such as text under black boxes or hidden columnsExtract all text from each file programmatically and run the stage 1 checks on itExtracted text is as clean as the record text
3 PixelsScreenshots, photos, scans, rendered pagesRendered names, address bars, notification pop-ups, faces, signatures, handwriting, plates, house numbersOCR-based detection, then a person views every kept image at full sizeNo identifier visible in any reviewed image
4 AudioVoicemail, call recordings, video soundtracksSpoken names, spelled-out emails, numbers read aloud, background voicesCompare redacted transcript spans with the audio and listen to a sample end to endEvery redacted span is silent or masked and no unflagged identifier is heard
5 Metadata and containerEXIF and XMP fields, document properties, PDF revision history, file names, email headers, archive listingsGPS coordinates, author and editor names, device owner names, prior versions, names in file namesRead all metadata with a metadata tool after every other edit is finishedNo original identifier in any metadata field, file name or prior revision
6 Cross-modal combinationThe record as a wholeDetails that identify only together, such as a storefront photo plus the ticket's city and dateA reviewer reads the processed record as an outsider wouldNo combination points to a person a motivated outsider could name
7 Sign-offAudit recordGaps, exceptions and open findingsPrivacy reviewer checks the stage log and samplesSigned record listing stages, samples, findings and fixes

Why does the order of the stages matter?

Order matters because later edits can create new exposure that earlier checks already passed. Redacting an image or a PDF writes a new file, and the tool that writes it may add or keep metadata. If you check metadata first, the clean result describes a file that no longer exists.

Presidio's documentation makes the same point for medical images: it recommends redacting text from DICOM pixels before scrubbing metadata. PDFs show why the container check must come last. ExifTool's documentation explains that it writes PDF metadata with an incremental update technique, so edits are reversible and old information is never actually deleted from the file; it suggests rewriting the file with the qpdf utility to remove the old information permanently. A PDF whose author field was edited may still carry the original author in an earlier revision.

Run stage 5 last, on the final files that will be delivered, and rerun it after any later change. ExifTool's documentation shows that the command option -all= deletes all meta information from an image, and also warns that not all groups are deletable and that some groups are not removed by default, so read the metadata back after stripping rather than assuming the strip worked.

What does each modality hide?

Each modality hides identifiers in predictable places, and reviewers find more when they know where to look. Use the list below as the reviewer's prompt for stages 2 to 5.

At a glance

What does each modality hide

Video combines every layer at once: frames that behave like screenshots or photos, a soundtrack that behaves like a call, captions that behave like text and a container with its own metadata. Treat a screen recording as a set of images plus an audio file, sample frames at a fixed interval as well as at every scene change, and expect the review to cost more than any single modality.

  • Screenshots: browser tabs and address bars that show email addresses in URLs, notification pop-ups, chat sidebars, the signed-in user's name in a corner, CRM or help desk record headers.
  • Photos: faces, reflections, house numbers, vehicle plates, name badges, paperwork on a desk, and GPS position in EXIF. ExifTool's GPS documentation lists GPSLatitude and GPSLongitude among the EXIF GPS tags.
  • Scans and signed forms: handwriting, signatures, stamps and initials in margins, which OCR often reads poorly or not at all.
  • PDFs: text under drawn black boxes, form field values, annotations, attached files and earlier revisions.
  • Spreadsheets and documents: hidden sheets, hidden rows and columns, comments, tracked changes and the author property.
  • Audio: names spelled letter by letter, numbers read in groups, callers who say who they are before the greeting, and other people speaking in the background.
  • Containers: file names such as a customer's name plus a date, folder paths inside archives, and email header fields beyond From and To.

How do you audit audio as well as transcripts?

Audit audio by checking that every span the transcript redacted is also silenced in the recording, then listening for identifiers the transcript never flagged. A clean transcript says nothing about the audio unless the audio was redacted from the same detections and checked afterward.

Some services produce redacted audio directly. Amazon Transcribe's documentation says that with Call Analytics a redacted audio file replaces spoken personal information with silence, and that this redacted media option is only supported for Call Analytics requests. The AWS service card for Amazon Transcribe cautions that PII detection may not identify all PII in all contexts and that customers should not rely solely on automated redaction without additional review.

The voice itself can identify a speaker, so silencing named spans does not make a recording anonymous. If the use does not need the audio, delivering reviewed transcripts only is the simpler answer, and that choice should be recorded as the outcome of stage 0.

Illustrative worked example: a ticket that looked clean

Illustrative example: a fictional property management software company prepares support tickets for licensing. Text redaction passed a random sample with no misses. The multimodal audit of one ticket type still found five exposures, listed in the table at the end of this section with the stage that caught each one; the three lessons come first.

  • Lesson one: the text sample result was accurate for text and silent about everything else.
  • Lesson two: three of the five fixes were exclusions, not redactions, because the attachment did not add enough value to the intended use to justify the review cost.
  • Lesson three: the combination finding came from a reviewer reading the processed ticket as an outsider would, not from any detector, so stage 6 needs a person with time set aside for it.
Illustrative worked example: a ticket that looked clean
LayerFindingStage that caught itAction taken
ScreenshotA tenant's full name and unit number in the header of the property manager's screen3 PixelsRegion masked, then the screenshot type set to exclude by default
PDF lease excerptTenant name present in the text layer under a drawn black box2 Embedded textPage re-rendered as an image, masked and text layer removed
Voicemail attachmentCaller spelled her email address; transcript redacted it but audio was untouched4 AudioVoicemail type excluded; transcript kept after review
Phone photo of a leakGPS coordinates of the building in EXIF5 MetadataAll metadata stripped and read back
Ticket as a wholeBuilding photo plus the city and date in the ticket identified one property6 CombinationExterior photos excluded; dates generalized to month

When is the audit not enough, and what are its limits?

The audit is not enough when the residual risk depends on what an outsider already knows, which no file check can measure. Stage 6 is a structured judgment, not a proof. Records about small communities, rare events or public figures need a separate re-identification review.

Automated detectors are also incomplete. Presidio's documentation says that because it uses automated detection there is no assurance it will find all sensitive information, and that additional systems and protections should be employed. Its image redactor is labeled beta. OCR struggles with handwriting, low resolution, rotated text and text over busy backgrounds, which is why stage 3 ends with a person viewing every kept image.

Utility is the other limit. Masking too much of a screenshot can remove the error message or setting that made the ticket useful, and silencing too much of a call can remove the customer's actual problem. Check a sample of processed records for usefulness as well as exposure, and when an attachment cannot be both safe and useful, exclude it rather than ship a version that is neither.

The main exception to the full sequence is a text-only delivery. If every non-text type is excluded at stage 0, stages 3 to 5 reduce to checking that no attachment, file name or embedded object slipped through. That is often the right scope for a first license.

Which audit findings should halt a release or need a specialist?

Stopping is a normal outcome of the workflow, not a failure of the team. Each stop should produce a short incident note: what was found, in which layer, how many delivered or staged files it could affect, the fix, and the rerun result. Keep those notes with the sign-off record so the next audit starts with the reviewer prompts already updated.

At a glance

Which audit findings should halt a release or need a specialist

  • Stop when a stage finds an exposure type you did not anticipate; update the reviewer prompts and rerun that stage on a fresh sample.
  • Narrow the scope when an attachment type repeatedly fails stage 3 or 4; exclude it and keep the text.
  • Ask a privacy specialist when images show people's faces, documents show signatures, recordings include minors, or records relate to health, finances or legal matters.
  • Ask counsel about recording consent and biometric rules before any audio or face image is licensed.
  • Pause delivery if metadata or a prior PDF revision exposes an identifier after sign-off, and rerun stage 5 on every delivered file.

Is a clean multimodal audit enough to license screenshots and recordings?

A clean multimodal audit answers a privacy question only. Whether you may license the records also depends on who owns them, what customers and staff were told, and what your contracts allow, and counsel reviews those facts for the specific dataset. De-identification alone does not create a right to license.

For tickets, scans and recordings licensed to AI developers through SourceX, the attachment decisions and layer checks belong to its data preparation work, which runs next to rights review and contracting, and the de-identification is carried out by SourceX through approved processing routes. Bulky image and audio sets do not have to be uploaded to a SourceX host: large deliveries stay in the seller's storage or travel on encrypted drives. Before any of that, the company's eligibility is decided in a brief assessment, and no image, recording or ticket has to be sent for it.

Frequently asked questions

Why did text redaction miss the name in the screenshot?

Text redaction reads characters stored as text. A screenshot stores the name as pixels, so the redactor never sees it unless the pipeline first runs OCR on the image and redacts the detected regions. Even then, a person should view each kept image, because OCR misses small, rotated or stylized text.

What should a multimodal privacy audit workflow include?

An inventory of file types, then checks of record text, text embedded in files, image pixels, audio, metadata and the record as a whole, in that order, with a sign-off record. Each stage has its own detector, its own human check and a written exit criterion.

Does stripping EXIF data remove all image metadata?

Not always. ExifTool's documentation notes that not all metadata groups are deletable and some are not removed by default, and other formats keep metadata elsewhere. Read the metadata back after stripping and check file names and container listings too.

Can a redacted PDF still contain the original text?

Yes. A black box drawn over text can leave the text layer intact, and tools that edit PDF metadata incrementally can leave earlier values in prior revisions. Extract all text from the final file and rewrite the file before delivery.

If the transcript is redacted, is the call recording safe to share?

No. The audio must be redacted from the same detections and checked, and the speaker's voice can still identify them. Many projects deliver reviewed transcripts only and leave recordings out.

Sources

  • Google's Sensitive Data Protection documentation says it uses optical character recognition to detect text within images and then analyzes that text, covering matches with opaque boxes. Source
  • The Presidio Image Redactor (beta) detects and redacts PII text in images using Tesseract OCR; its DICOM engine redacts only pixel data, not metadata, and it recommends redacting pixels before scrubbing metadata. Source
  • Presidio's documentation says automated detection may not find all sensitive information and additional systems and protections should be employed. Source
  • ExifTool writes PDF metadata with an incremental update technique, so edits are reversible and old information is never actually deleted; qpdf can remove it permanently. Source
  • ExifTool's documentation shows -all= deleting all meta information from an image and notes that not all groups are deletable and some are not removed by default. Source
  • ExifTool's GPS tag documentation lists GPSLatitude and GPSLongitude among EXIF GPS tags stored in a separate IFD. Source
  • Amazon Transcribe documentation says a redacted audio file from Call Analytics replaces spoken personal information with silence, and redacted media is only supported for Call Analytics requests. Source
  • The AWS AI Service Card for Amazon Transcribe cautions that PII detection may not identify all PII and advises against relying solely on automated redaction without additional review. Source
  • NISTIR 8053, De-Identification of Personal Information (2015), discusses the challenges of de-identifying non-tabular data such as free-format text and images. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies