Privacy and preparation
Dataset record deletion selection bias: what survivors really show
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Assess deletion-driven selection bias by defining the population the model will serve, listing every deletion path and its trigger, counting what each path removed by stratum from traces such as ID gaps, schedules and old reports, and comparing survivors with those counts. Weight only gaps driven by observed fields, disclose content-driven gaps, and state which conclusions the remaining dataset cannot support.
Redaction
Dataset record deletion
Key takeaways
- Deletion is a selection process: each path has a trigger, and the trigger predicts which records are missing.
- Age-based schedules can be corrected by weighting; deletion driven by the content of the removed records cannot.
- Count removed records from content-free traces such as ID gaps, schedules, logs and old reports, never by restoring erased data.
- Compare survival ratios by stratum against the population the model will serve, not against the archive.
- Ship a one-page deletion and selection limitation disclosure that names the claims the data should not support.
How do you assess coverage changes caused by deleted records?
Start by defining the population the model will serve, then reconstruct every path by which records left the archive, count what each path removed by stratum, and compare the surviving records with an independent count of what once existed. Where the gap is uneven across customers, periods, channels or outcomes, the remaining dataset no longer shows what the business actually did, and the dataset's documentation has to say so.
Deletion is a selection process. Retention schedules remove old records, privacy requests remove particular people, administrators remove spam, test entries and embarrassments, and system migrations drop whatever did not map. Each path removes a different kind of record, so the survivors can show higher resolution rates, fewer complaints or a younger customer base than reality without a single surviving record being wrong.
The output of this work is a short disclosure that travels with the dataset: what the population is, which deletion paths ran, what they removed, how large the gaps are where they can be measured, and which conclusions the data should not be used to support. The rest of this page builds that disclosure step by step and closes with a template.
Which deletion paths skew a dataset, and in which direction?
Every deletion path has a trigger, and the trigger predicts the skew. Age-based schedules thin out early periods. Person-initiated erasure removes the people most likely to have had a bad experience or to have left. Administrative cleanup removes records someone considered noise, which sometimes really is noise and sometimes is the hard cases.
Privacy law creates several of these paths. Article 17 of the GDPR gives people a right to erasure on listed grounds, including where the data is no longer necessary for its purpose, where consent is withdrawn and no other legal ground applies, or where the person objects; it also lists exceptions. California's CCPA lets consumers ask a business to delete personal information collected from them, subject to exceptions such as information the business must keep by law. Who uses those rights is not random, and that is the point for coverage.
| Deletion path | Typical trigger | What it tends to remove | Likely direction of skew |
|---|---|---|---|
| Age-based retention schedule | Records older than a set period after closure | Early years, long-closed cases | Survivors over-represent recent periods and current processes |
| Person-initiated erasure | Privacy request or account closure | Records of customers who left or complained | Survivors look more satisfied and more loyal |
| Administrative cleanup | Spam, test, duplicate or abandoned records | Noise, but also unusual valid cases | Fewer edge cases; sometimes a cleaner, more accurate population |
| Deal or project closure purge | Lost deals, cancelled jobs, withdrawn applications | Failures and non-starts | Survivors overstate win and completion rates |
| Migration loss | Fields or objects that did not map to a new system | Whole record types or one era's attachments | Step change in coverage at the migration date |
| Legal hold exemption | Records kept because of a dispute | Nothing; it retains a disputed subset | Survivors in purged periods over-represent disputes |
Can the deletion be ignored? Three questions borrowed from missing-data theory
Whether you can ignore a deletion depends on what drove it, not on how many records it took. Statistical work on missing data, starting with Donald Rubin's 1976 paper in Biometrika, set out the conditions under which the process that causes missing data can be ignored for inference. Applied to deleted records, three practical questions follow.
First, was the deletion unrelated to anything in the data, such as a storage outage that removed a random block? Then survivors are a smaller but fair sample. Second, was it driven by something you can still observe, such as record age or region? Then you can reweight or stratify by that variable, provided the deletion rule is documented. Third, was it driven by the content of the deleted records themselves, such as customers with bad outcomes asking to be forgotten? Then no amount of analysis on the survivors recovers the missing pattern, and the disclosure must limit the claims.
Most real archives mix all three. The work is to assign each deletion path to one of these questions, using its documented trigger, and to say which conclusions each one puts out of reach.
Counting what is gone: where deleted-record traces survive
You usually cannot see deleted records, but you can often count them. Systems leave traces: sequential identifiers with gaps, audit logs, recycle bins, schedule configurations and reports produced before the purge. These give a denominator without bringing back any deleted content.
At a glance
Counting what is gone: where deleted-record traces survive
Platform behavior determines how long those traces last. Salesforce documents that, by default, deleted items stay in the Recycle Bin for 15 days before they are scheduled for permanent deletion. Zendesk deletion schedules act only on archived tickets, which are tickets closed for more than 120 days, delete tickets that stayed closed and unmodified for the chosen period, and cannot be restored. A schedule's configuration is itself evidence: its conditions say which tickets were removed and from when.
Do not reconstruct the content of records deleted because a person asked. The goal is a count by stratum, never a recovery of the erased data, and a counsel review should confirm which traces may be used at all.
- Sequential IDs: gaps in ticket, order or invoice numbers by month give a removed count.
- Deletion schedules and retention settings: the configured conditions and activation dates.
- Audit or event logs: delete events with object type and date, without content.
- Historic aggregate reports: monthly volumes or outcome rates produced before purges ran.
- Finance and ledger records: invoices or payments that imply a case existed even if its ticket is gone.
- Privacy request logs: counts of erasure requests by month, kept without the requesters' identities in the dataset.
Comparing survivors with the population the model will serve
Compare by stratum, not in total. A dataset can retain 90 percent of its records overall and 40 percent of one customer segment. Choose strata that matter to the model's use: period, channel, customer type or size, region, product, outcome, and record age at deletion time.
For each stratum, compute a survival ratio: surviving records divided by the independent count. Then look for three patterns. Step changes at schedule or migration dates show period gaps. Ratios that fall with outcome severity show content-driven deletion. Ratios that differ by segment show who is underrepresented.
Datasheets for Datasets, a widely used documentation framework, asks creators whether a dataset is a sample of a larger set, whether it is representative, and whether information is missing from instances and why. A deletion analysis is the evidence behind those answers, and stating the survival ratio by stratum is more useful to a buyer than a single sentence saying the data is representative.
The deletion policy and selection limitation disclosure
The disclosure is one page that answers what was removed, why, how much, and what that rules out. The template below is editorial, version 1.0 dated 2026-10-18; entries are illustrative and describe no real dataset. Fill it before release and update it whenever a new deletion path runs on the source system.
At a glance
The deletion policy and selection limitation disclosure
| Field | What to write | Illustrative entry | Evidence behind it |
|---|---|---|---|
| Deployment population | Who and what the model will serve | Inbound support requests from small business customers, all channels | Product and channel scope from the buyer's request |
| Archive window | Earliest and latest record dates in the delivery | 2021-01 to 2026-06 | Min and max creation dates in the extract |
| Deletion paths | Each path, its trigger and its active dates | Ticket deletion schedule from 2024-03; erasure requests throughout | Schedule settings and policy documents |
| Removed counts by stratum | Independent count, surviving count, survival ratio | 2021 email tickets: survival ratio 0.38 | ID gaps and pre-purge monthly reports |
| Mechanism classification | Unrelated, driven by observed fields, or driven by content | Schedule: by age (observed); erasure: likely content-related | Trigger definitions, analyst review |
| Adjustments applied | Weights, stratified sampling or none | Weights by year and channel for the age schedule only | Weighting script and version |
| Underrepresented groups | Segments with low survival relative to the population | Churned customers; complaints escalated to refunds | Survival ratios by segment and outcome |
| Claims not supported | Conclusions the data should not be used for | Customer satisfaction rates; churn prediction | Mechanism classification |
| Out-of-population removals | Deletions that improved fit | Spam and internal test tickets removed | Cleanup criteria and counts |
| Review and refresh | Who reviewed and when to redo the analysis | Statistician and support operations lead; on any new schedule | Sign-off record |
Worked example (illustrative): support tickets after two purges
This example is illustrative; the company, counts and ratios are invented to show the method.
A support team's archive runs from 2021 to mid-2026. Two deletion paths ran: a schedule activated in 2024 that removes tickets closed and untouched for two years, and a steady trickle of erasure requests, which the team's privacy log shows came disproportionately from customers who closed their accounts. Ticket IDs are sequential, and monthly volume reports from the old dashboard survive for every month.
The analyst counts by year and channel. Survival is near 1.0 from 2024 on, falls to about 0.4 for 2021 to 2022 email tickets, and the 2023 phone queue sits in between. Within every year, tickets linked to accounts that later closed survive at a visibly lower rate than tickets from active accounts. The age-driven gap is explainable by an observed variable, so the analyst weights by year and channel. The erasure-driven gap is tied to outcomes, so no weight fixes it.
The disclosure therefore supports training a classifier that routes tickets by topic, with year weights, and states that resolution and satisfaction rates computed on the data will look better than reality because the unhappiest customers are underrepresented. It rules out using the set to predict churn.
The exception: the team also removed about 6 percent of all tickets as spam and internal tests. Those records were never in the deployment population, so this deletion improves coverage. It still belongs in the disclosure, with its criteria, so a buyer can tell it apart from the other two paths.
Who ends up underrepresented relative to the intended users?
Compare survival ratios against the intended users, not against the archive. If the model will serve every customer, the groups to name are the ones whose records were removed at higher rates: people who left, people who complained, early-era customers whose processes have since changed, and any channel retired before the retention schedule began.
NIST's work on bias in AI identifies datasets as one of the main challenge areas for managing bias, alongside testing and evaluation and human factors. A deletion analysis is a concrete way to meet that challenge, because it turns a general worry about representativeness into named groups with measured gaps. Where a group cannot be measured, the disclosure should say that rather than leave the impression that it is covered.
Narrowing the claim when a deletion cannot be measured
If you cannot count what a path removed, limit the use rather than guess the size. Drop the affected stratum, restrict claims to periods after the path started, or describe the data as a sample of surviving records with an unknown selection mechanism. Each option is honest; a confident weight built on an unmeasured gap is not.
Bring in a statistician when a content-driven deletion touches the outcome the model predicts, when weights change a headline rate by a margin that would change a decision, or when the buyer plans to report population-level figures. Bring in counsel when any trace used for counting comes from records deleted at a person's request, or when the deletion history itself might reveal who asked.
Disclosing deletion when records are licensed for AI
Deletion history should be disclosed in the same document that describes the data, not discovered by a buyer after training. A buyer can reasonably ask for the disclosure above before relying on any rate the data appears to show, and a supplier can prepare it from settings and logs without exposing deleted content.
Buyers working with SourceX describe scope, volume, format, timing and licensing needs in a request, and the records are then sourced from U.S. businesses rather than drawn from stock; a population definition and a deletion disclosure fit naturally into that request. The supplying company approves the scope and signs the agreement before any records are delivered, which is the point at which known deletion paths and their limits should be written down.
Frequently asked questions
What should the deletion policy and selection limitation disclosure include?
The deployment population, the archive window, each deletion path with its trigger and dates, removed and surviving counts by stratum, a classification of each path's mechanism, any adjustments applied, the groups left underrepresented, the claims the data should not support, and who reviewed it. Keep it to about a page so it is actually read.
Does deleting records always bias a dataset?
No. Deletion that is unrelated to the data, or that removes records outside the population such as spam and test entries, can leave a fair or even better sample. Bias arises when the reason for deletion relates to what the model will learn, such as customers with bad outcomes being more likely to be erased.
Can weighting fix gaps caused by deleted records?
Only when the deletion was driven by something still visible in the data, such as record age or channel, and the rule is documented. When deletion depended on the content of the removed records themselves, survivors carry no information about what is missing and weighting cannot recover it.
How can you count deleted records without restoring them?
Use traces that hold no content: gaps in sequential identifiers, delete events in audit logs, deletion schedule settings, privacy request counts and aggregate reports produced before the purge. Never reconstruct records a person asked to erase.
When should we stop or narrow the dataset's intended use?
Narrow it when a content-driven deletion touches the outcome the model predicts, or when a removed count cannot be measured. Restrict claims to unaffected periods or segments, and involve a statistician and counsel before reporting population-level rates.
Sources
- GDPR Article 17 gives data subjects a right to erasure on listed grounds, such as data no longer being necessary, withdrawn consent or a successful objection, with stated exceptions. Source
- Under the CCPA, consumers may ask a business to delete personal information collected from them, subject to exceptions such as information the business is legally required to keep. Source
- Rubin's 1976 Biometrika paper sets out the weakest general conditions under which the process that causes missing data can be ignored when making inferences. Source
- Datasheets for Datasets proposes documentation questions including whether a dataset is a sample of a larger set, whether it is representative, and whether information is missing from instances. Source
- Salesforce keeps deleted items in the Recycle Bin for 15 days, after which they are scheduled for permanent deletion. Source
- Zendesk ticket deletion schedules delete only archived tickets (closed for more than 120 days) that stay closed and unmodified for a set period, and deleted tickets cannot be restored. Source
- NIST SP 1270 identifies datasets, testing and evaluation, and human factors as key challenge areas for managing bias in AI. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.