Skip to content

Privacy and preparation

De-identified dataset residual risk report: what it must say

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A residual risk report for a de-identified dataset should state the release it covers, the transformations applied, the tests run and results, and a register of methodology limitations and unresolved risks, each with a status, an owner, the control it relies on and a trigger for reassessment. It should name the recipient assumptions behind the conclusion and never claim that the remaining risk is zero.

Redaction

Deidentified dataset

This page covers de-identified dataset residual risk report: what it must say.Illustrative

Key takeaways

  • The residual report states what risk remains for a specific recipient and use, not just which techniques were applied.
  • Address singling out, linkage and inference separately, and test each modality in the files, including notes, attachments, media and timestamps.
  • Keep one register entry per limitation or open risk, each with a status, owner, relied-on control and reassess trigger.
  • Pair every privacy measure with its task utility cost so nobody removes a protection without knowing why it exists.
  • Hold the release when an open risk has no owner or the conclusion depends on a control the agreement lacks.

What should a residual privacy risk report for a de-identified dataset contain?

A residual risk report should state what risk is left after de-identification, for which recipient and environment, under which assumptions, and what was not tested. It needs five parts: the release it covers, the transformations applied, the tests run and their results, a limitations and unresolved risks register, and the conditions the conclusion depends on. The register is the part most reports skip and the part a reviewer reads first.

Metrics and methods matter, but they answer a narrower question: did the techniques work on what was measured? The residual report answers the question a licensing decision actually turns on: given everything that was and was not checked, is the remaining risk acceptable for this recipient and this use, and what would change that answer?

Write it for two readers at once. A privacy specialist needs enough method detail to reproduce the reasoning. A data owner signing a license needs a plain statement of what could still go wrong and who is watching for it.

Why can no report honestly claim the risk is zero?

De-identification reduces disclosure risk while trying to keep the data useful, so some risk always remains and the report's job is to describe it accurately. NIST SP 800-188, the agency guidance on de-identifying government datasets published in September 2023, frames de-identification as preventing or limiting disclosure risks while still allowing meaningful statistical analysis. Limiting is the operative word.

Health privacy guidance makes the same point in its own terms. HHS guidance on the HIPAA Expert Determination method says there is no explicit numerical level of identification risk that universally meets the very small standard, and that the methods and results of the analysis must be documented. The level depends on the anticipated recipient. That is a residual risk judgment, written down.

Tooling has limits as well. The Microsoft Presidio documentation says that because the project relies on automated detection, it cannot promise to find all sensitive information, and that additional systems and protections should be used. A report that cites a detector's output without stating its blind spots is overstating its own evidence.

How much method detail belongs around the register?

Enough that a second assessor could rerun the work and reach the same residual statement, and no more. The surrounding parts of the report are short by design; their job is to bind the register to one exact release and make each entry checkable.

At a glance

How much method detail belongs around the register

Keep each part to what a reviewer will actually verify. Long narrative descriptions of techniques that are standard add little; the parameters, versions and samples that make this release different add a lot.

  • Release binding: dataset name, version, extraction window, record and file counts, and file hashes, so the report cannot be reused for a later delivery by accident.
  • Transformations: each step in the order it ran, with tool names, versions and parameters, such as the date-shift range, generalization levels and surrogate rules.
  • Tests and results: what was measured, on which sample, by whom, with the raw counts behind any rate, and a pointer to where the working files are kept.
  • Register: the limitations and unresolved risks entries described below, each with a status.
  • Conditions and sign-off: the recipient controls the conclusion depends on, the assessor's name and independence, and the date after which the report should not be relied on without review.

Which identifying combinations and modalities can survive the transformation?

Removing names and account numbers leaves three routes to identification, and the report should address each one by name. The Article 29 Working Party's Opinion 05/2014 on anonymisation techniques describes them as singling out a person's records, linking records about the same person within or across datasets, and inferring an attribute's value from other attributes. A technique can block one route and leave another open.

Business records add a modality problem. A CRM export is not one table; it carries free-text notes, email threads, attachments, call recordings, photos and timestamps, and each of these needs its own test. The table below is a starting checklist of residual pathways in operational records.

Which identifying combinations and modalities can survive the transformation?
Residual pathwayWhere it hides in business recordsTest that addresses itIf untested, the report says
Rare combinations of kept fieldsJob title plus site plus hire month in HR or field service dataEquivalence class counts on agreed quasi-identifiersCombinations not measured; recipient joins prohibited
Identifiers inside free textNames and phone numbers typed into ticket notesDetector run plus a hand-labeled recall sampleRecall unknown; residual rate not estimated
Linkage across systemsTicket timestamps that match public status pages or social postsLinkage simulation against plausible outside sourcesOutside sources considered, linkage not attempted
Attribute inferenceProduct mix revealing a customer's industry or sizeInference test on sensitive attributesInference not assessed for named attributes
Embedded files and mediaSignatures in scanned PDFs, faces in site photos, voices in callsModality-specific review or exclusionModality excluded or not reviewed
Release-over-release differencingMonthly deliveries where one new record stands outDiff analysis across planned releasesOnly this release assessed

What should the methodology limitations and unresolved risks template include?

Use one register entry per limitation, written so each entry can be accepted, mitigated or escalated on its own. A limitation is anything the assessment did not cover or could not measure well; an unresolved risk is a known pathway that remains open after the transformation. Both go in the same register because both qualify the conclusion.

Each entry should end in a decision with a named owner. An entry that only describes a gap, with no status, is a note; an entry with a status, owner and review trigger is a control.

If the dataset ships with a card, the register's summary belongs in it. The Hugging Face dataset card template has a section for bias, risks and limitations covering technical and sociotechnical limits, and the Croissant responsible AI vocabulary has machine-readable properties for data limitations and for personal or sensitive information. Point those fields at the register rather than paraphrasing it, so the card and the evidence cannot drift apart.

What should the methodology limitations and unresolved risks template include?
FieldWhat to writeIllustrative entryOwner
Entry ID and releaseStable ID tied to the dataset version and file hashesL-04, release 2Assessor
TypeMethodology limitation or unresolved riskUnresolved riskAssessor
PathwaySingling out, linkage, inference, modality or differencingLinkage through free-text addressesAssessor
What was doneTests run, sample sizes, tools and versionsDetector pass plus a hand review of a labeled sampleAssessor
What was not done, and whyScope, time, tool or access reasonNo review of scanned attachments; excluded insteadAssessor
Effect on the conclusionHow the gap could change the residual risk statementCould raise risk for small rural customersPrivacy specialist
Utility cost of closing itWhat fixing it would remove from the taskDropping addresses breaks routing evaluationTask reviewer
StatusAccepted, mitigated, open, or blocks releaseMitigated by generalizing to regionPrivacy specialist
Control relied onContract term or environment control the status depends onNo linkage to outside address dataLegal reviewer
Reassess whenEvent that reopens the entryNew field added or new recipientData owner

How should privacy risk and task utility appear side by side?

Report them together, because every privacy measure has a task cost and a buyer cannot weigh one without the other. Date shifting can break service level durations. Generalizing locations can erase the routing signal a dispatch model needs. Replacing names with surrogates keeps conversation flow but can leave relationships between records intact. A report that shows only privacy results invites the next person to undo a protection to restore utility, without knowing what it was for.

A practical format is a paired line for each transformation: what it protects against, what task signal it removes or distorts, and how that was checked. The check can be simple, such as confirming that durations computed from shifted dates match durations from originals, or that category distributions survive generalization.

When a protection and a task need conflict, the register records the trade and who made it. Sometimes the honest answer is that a field cannot be both safe enough and useful enough, and it leaves the scope.

Ask the buyer's task reviewer to name the two or three signals the model most depends on before preparation starts, not after. A support routing model may need queue, product and resolution time but not location; a dispatch model may need location at some resolution but not customer type. Knowing this early lets the assessor spend the privacy budget where it costs the task least, and it gives the register a concrete utility column instead of a guess.

Worked example: residual risk entries for field service records (illustrative)

This example is invented to show the method; it describes no real company or dataset. A regional HVAC services firm prepares three years of ServiceTitan-style work orders for licensing: job type, equipment, dates, technician notes, and photos. The preparation removes customer names, phone numbers and street addresses, keeps city and ZIP3, shifts dates per customer, and runs a detector over technician notes.

The assessor's register ends up with six entries. Two are worth showing. Entry L-02 records that technician notes were checked with a detector and a hand-labeled sample, but the sample was drawn only from the most recent year, so recall on older notes with different writing habits is unknown. Status: open, with a second sample scheduled before release. Entry L-05 records that equipment photos were not reviewed frame by frame; instead, all photos showing house exteriors were excluded and the remainder reviewed in a sample. Status: mitigated, with the exclusion counted in the exclusion log.

The meaningful exception is a third entry. In two small towns in the service area, ZIP3 plus job type plus equipment model singled out individual households in the assessor's counts. Generalizing further would have erased the regional climate signal the task needed. The firm removed those towns from scope rather than shipping a known singling-out risk, and the register says so.

Which recipient assumptions does the residual risk conclusion rest on?

Every residual risk statement assumes something about who receives the data and what they can do with it, and the report should list those assumptions as conditions. The UK ICO's anonymisation guidance uses a motivated intruder test and treats the release context as part of the identifiability question: releasing to the world at large calls for more robust techniques than releasing to a controlled recipient.

Typical conditions for a licensed business dataset are a ban on re-identification attempts, a ban on linking to named outside sources, access limited to a named team in a controlled environment, retention limits, and notice before onward transfer. These become contract terms, so the legal reviewer should read the conditions list against the draft agreement line by line.

Recipient changes reopen the assessment. HHS guidance notes that future releases of data to the same recipient may need additional or different de-identification. The same logic applies to a new recipient, a new field, or a new delivery cadence, which is why every register entry carries a reassess-when trigger.

When should an open finding block the release or go to a specialist?

Hold the release when an unresolved risk has no mitigation and no accepted owner, when a pathway was untested for a modality that is present in the files, or when the conclusion depends on a control the draft agreement does not contain. Each of these means the report cannot support the statement it is meant to make.

Bring in a specialist when the data includes health, financial, biometric or children's information, when the records describe a small population where singling out is likely, when the buyer intends to combine the data with other sources, or when the transformation relies on statistical methods such as differential privacy whose parameters need expert interpretation. These are general triggers, not legal thresholds; counsel and a privacy specialist decide what applies to the specific records.

Narrowing the scope is often the cleanest fix. Dropping a modality, a small geography or a rare category usually costs less utility than a stronger transformation across the whole dataset, and it turns an open entry into a closed one.

Whatever the outcome, record the decision in the register itself: who decided to hold, narrow or proceed, on what date, and on what evidence. A residual report that ends with a dated decision is the document a privacy specialist can defend later; one that ends with a list of caveats leaves the decision to whoever reads it next.

Residual risk reporting when SourceX prepares the dataset

In a SourceX licensing deal, de-identification is carried out by SourceX through approved processing routes, so the residual risk report and its register belong with the preparation evidence the company reviews. Because the company signs off on scope and terms before any delivery, the signing stage is when its reviewers should read the limitations register and match each condition to an agreement clause. Qualification is checked first, through a brief assessment that involves no data transfer.

Frequently asked questions

Is a residual risk report the same as a re-identification risk assessment?

It is the conclusion of one. The assessment covers methods, metrics and tests; the residual report states what risk remains, what was not tested, which assumptions it rests on and who owns each open item. Many assessments include a short residual section, but a licensing decision needs it written out in full.

Who should write the limitations section?

The assessor who ran the tests drafts it, because only they know what was skipped and why. A task reviewer adds the utility cost of each fix, and a legal reviewer confirms that every control the conclusion relies on appears in the agreement.

Does an accepted residual risk mean the data is legally de-identified?

Not by itself. Legal standards differ by law and jurisdiction, and some require specific methods, documentation or contractual commitments. The report is evidence for that legal judgment, which counsel makes on the specific facts.

How often should the report be updated?

Whenever an entry's reassess trigger fires: a new field, a new modality, a new recipient, a new delivery cadence or a new outside data source that makes linkage easier. For recurring deliveries, review the register before each release even if nothing appears to have changed.

Can a residual risk report include numbers like k values or detector recall?

Yes, when they were measured on the release in question and the report states the sample and method. What it should not do is present a single number as the whole answer, because the untested pathways in the register often matter more.

Sources

  • HHS guidance on HIPAA Expert Determination says no single numerical risk level universally meets the very small standard, that methods and results must be documented, and that future releases may need re-examination. Source
  • NIST SP 800-188, published September 2023, describes de-identification as preventing or limiting disclosure risks while allowing meaningful statistical analysis. Source
  • The ICO's anonymisation guidance uses the motivated intruder test and treats release context, from controlled recipients to the world at large, as part of identifiability. Source
  • Article 29 Working Party Opinion 05/2014 assesses anonymisation techniques against singling out, linkability and inference. Source
  • Microsoft Presidio's documentation says automated detection cannot find all sensitive information and that additional protections should be employed. Source
  • The Hugging Face dataset card template includes a bias, risks and limitations section for technical and sociotechnical limitations. Source
  • The Croissant RAI vocabulary defines properties for dataset limitations and for personal and sensitive information. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies