Privacy and preparation
Market research sample replacement: how swaps change coverage claims
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A study should account for replaced respondents by logging every replacement before the data leaves the agency: the rule that allowed it, the stage, the reason, the quota cell, the source of the substitute and the date, under study-scoped pseudonymous IDs. Report the number and nature of replacements, and restate the coverage claim as achieved rather than as designed whenever replacements cluster in particular cells or sources.
License terms
Market research sample
Key takeaways
- Every replacement is an event with a rule, a stage, a reason, a cell, a source and a date.
- Matching a quota cell does not show that a replacement was neutral; clustered swaps move the claim.
- AAPOR requires substitutions to be reported, and its Code asks for quotas and replacements to be described.
- The replacement log is respondent data, so it uses pseudonymous IDs, day-level dates and fixed reason codes.
- Rewrite the claim as achieved, not as designed, before a study's records are reused or licensed.
How should a study account for replaced respondents in its coverage claims?
It should treat every replacement as an event with evidence, and let the coverage claim follow the evidence. A study designed as a national quota sample of 1,000 adults and delivered as 1,000 completes can still describe a different population if a large share of completes were swapped in late, from a different panel, into a few cells. The sentence in the report should describe what was achieved.
Survey research standards already require this for probability samples. AAPOR's Standard Definitions say that any substitution of sampled cases, replacing an originally sampled unit with another, must be reported, and the earlier edition spells out three parts: document the substitution rules, report the number and nature of substitutions, and account for every replaced case in the final disposition codes. The same discipline works for quota and panel studies, where the claim rests on how the sample was assembled.
This matters more when a study's records are reused. When an agency licenses respondent-level data for AI training or evaluation, the buyer inherits the coverage claim. A claim that no longer matches the records travels with the dataset and misleads every model built on it.
Where replacement happens in a market research study
Replacement is not one event. It happens at several stages, for different reasons, and each changes the sample in its own way. Agencies usually keep the evidence for some of these in the survey platform, some in the panel provider's reports and some only in a project manager's email.
Start by listing which of these stages a study actually went through, then find where each one left a trace. Survey platforms usually hold disposition and quota status data for each case, panel providers send completion and reconciliation reports, and the data processing team keeps the cleaning script that applied the quality checks. Pull those into one place before anyone writes the methods note, because they are hard to reassemble a year later.
| Stage | Typical trigger | What it can change | What to log |
|---|---|---|---|
| Fieldwork | Invitee does not respond; cell filled from fresh invites | Who answers: quicker, more active panelists | Invite wave, cell, source, date filled |
| Quality removal | Speeding, straight-lining, failed attention checks | Cells where removed cases were concentrated | Check failed, count by cell, backfill source |
| Quota management | Cell closes early or overfills; late cells backfilled | Balance between early and late responders | Cell close dates and backfill counts |
| Source blending | Second panel or third-party supplier added mid-field | Mix of recruitment methods within cells | Supplier per case, date added, client notice |
| Tracker attrition | Wave respondents drop out; refreshment sample added | Comparability between waves | Retained versus fresh cases per wave |
| Qualitative recruiting | No-show replaced from the screener backlog | Fit against screener criteria | Original and replacement screener answers |
| After fieldwork | Withdrawal or deletion request | Delivered counts and cell totals | Request date, cases removed, files updated |
Why the claim moves even when every quota still adds up
Quotas control the traits you set them on, usually age, gender, region and perhaps income. A replacement that fits the cell can differ on everything else: how fast the person answers, how many surveys they take, which panel recruited them. When replacements cluster, those unmeasured differences cluster too.
The history of quota sampling is a reminder. AAPOR's 2013 task force report on non-probability sampling recalls the 1948 U.S. election, where the major polls relied on quota sampling and called the winner wrong, and the review that followed found that interviewers had tended to pick somewhat more educated and better-off people within their quotas. The report notes that today's nonprobability panel samples are usually quota samples. The lesson for a replacement log is simple: matching the cell does not show that the swap was neutral.
In practice, three patterns deserve a closer look. Backfills that arrive in the last days of field come from people who were slower to respond or were invited later. Backfills from a different supplier come from people recruited by different methods and incentives. Backfills concentrated in one or two cells mean those cells now describe a different mix of people from the rest of the sample. None of these is wrong in itself; each one has to be visible in the claim.
Which disclosure rules already cover replacements?
Several published standards ask for pieces of this evidence, so the brief below mostly gathers what a careful agency should already hold. AAPOR's Code of Professional Ethics and Practices asks researchers to describe the sample design and any use of quotas, and says any data imputation, exclusions or replacement will also be discussed. AAPOR's report on evaluating survey quality lists whether respondents such as speeders or satisficers were removed from the final dataset among the questions a study should answer.
For online samples, ESOMAR's questions for buyers of online samples ask how multiple sample sources are blended, how duplicate respondents across sources are handled, and whether clients are told in advance when a third-party source is used. ESOMAR now publishes a 37-question edition of that guide. ISO 20252 sets vocabulary and service requirements for market, opinion and social research; ISO lists the 2019 edition as withdrawn and replaced by ISO 20252:2026, so check the current text for the clauses your certification body audits.
The privacy-reviewed sampling and change evidence brief
Prepare one brief per study, or per wave for a tracker, before any respondent-level data is licensed or shared. Each row names its evidence and its privacy treatment, because a replacement log is itself respondent data. The ICO describes data minimization as holding personal data that is adequate, relevant and limited to what is necessary, and warns against keeping data on the off-chance it might be useful; the log should carry reason codes and pseudonymous IDs, not names, emails or panel member numbers.
At a glance
The privacy-reviewed sampling and change evidence brief
| Brief field | What it contains | Privacy treatment | Signed by |
|---|---|---|---|
| Design statement | Target population, frame, mode, sources and planned quotas | No personal data | Project lead |
| Replacement rules | Rules for backfill, removal and blending, dated before field | No personal data | Project lead |
| Replacement log | Case ID, stage, reason code, cell, source, date | Study-scoped pseudonymous IDs; day-level dates | Field manager |
| Quality removals | Checks used, thresholds, counts removed per cell | Counts only in the brief; case list kept internal | Data processing lead |
| Source mix | Share of completes per supplier, per cell, with client notice | Supplier names are business data, not respondent data | Sample manager |
| Weighting | Variables, targets and their sources, or a statement of no weights | Aggregate targets only | Analyst |
| Achieved coverage | Rewritten claim describing the delivered sample | No personal data | Research director |
| Withdrawals and deletions | Requests received, cases removed, versions updated | Request references, not identities | Privacy lead |
| Minimization review | Fields dropped from the log and why each kept field is needed | Record of the review itself | Privacy lead |
| Reuse check | Notice and contract terms that cover the intended reuse | Clause references only | Rights reviewer |
Worked example (illustrative): one wave of a consumer tracker
The counts below are invented to show the method; they describe no real study. A tracker wave targets 1,000 completes of U.S. adults with quotas on age, gender and region from one main panel. During field, the agency removes 84 completes for failing quality checks and backfills them. The two youngest male cells run short, so the sample manager adds a second panel for the last three days, which supplies 61 of the backfills. After delivery, 6 respondents ask to be deleted.
- Designed claim: a nationally representative sample of 1,000 U.S. adults.
- Achieved claim (illustrative): a quota sample of U.S. adults matched on age, gender and region; 994 completes after 6 deletions; 84 quality replacements, 61 of them in two younger male cells from a second panel added late in field.
- Brief entries: replacement log under wave-scoped IDs, check thresholds, the supplier change with the date the client was told, and a deletion record for each affected file version.
| Event (illustrative) | Count | Where it landed | Effect on the claim |
|---|---|---|---|
| Quality removals backfilled | 84 | Spread across cells, heavier among younger respondents | Report removals and checks used |
| Backfills from a second panel | 61 | Two youngest male cells, final three days | Name blending; claim no longer single-source |
| Cells closed early | 5 | Older respondent cells | Older cells are early responders only |
| Post-delivery deletions | 6 | Mixed cells | Delivered total and cell counts change |
The exception: removals that never belonged in the sample
Some removals do not move the coverage claim. Duplicate completes, bots and respondents who were never eligible were not valid members of the sample, so removing them restores the design rather than changing it. Log them anyway with a reason code, because a buyer cannot tell a fraud sweep from a quota rescue unless the log says which is which.
Qualitative recruiting is a second case. A replaced interviewee in a 20-person study does not change a statistical coverage claim, because there is none; what matters is whether the replacement met the same screener criteria. Keep both screener records, pseudonymized, so the fit can be checked later.
Which service condition or record shows the obligation was met?
Two different obligations are in play, and they are met by different evidence. The client's specification, usually in the proposal or statement of work, sets counts, quotas, incidence, quality checks and sometimes approved sources; the field disposition file and the replacement log show whether those were met. The coverage claim in the report is a separate statement about the population, and only the achieved-coverage row in the brief can support it.
Keep the link between them explicit. If the client approved a second panel by email, attach that approval to the source mix row. If a deletion request arrived after delivery, record which delivered files were reissued. A dataset reviewer should be able to trace each change from the claim back to a dated record without reading anyone's inbox.
When the specification and the claim disagree, the specification usually decides whether the agency delivered what was bought, and the claim must be narrowed to match. An agency that met every quota with late backfills from a second source has met its service obligation and still owes the reader an accurate description of the sample.
When to hold a study out of a licensed dataset
Hold a study back, or license it only with a narrowed description, when its sample history cannot be reconstructed or when the log itself raises privacy questions. A qualified survey methodologist should read the brief for any study whose coverage claim will be repeated to a buyer, and a privacy or rights reviewer should confirm that respondent notices and client contracts allow the reuse at all; de-identification alone does not create that permission.
- Replacement rules were written after fieldwork, or not at all.
- Sources were blended without a record of which cases came from where.
- Reason codes are free text that describes individuals.
- Deletion or withdrawal requests cannot be traced to the delivered files.
- The client contract or respondent notice does not cover the intended reuse.
- The achieved sample differs from the claim and nobody will sign the rewrite.
What a licensing review asks about sample history
A licensing review asks for the brief, not the respondents. SourceX licenses business data directly from agencies and other companies, and an agency's first step is a short company assessment with no data upload. The agency approves the scope and terms and signs before any respondent-level file is delivered, so studies with incomplete sample histories can be left out of scope from the start.
Frequently asked questions
What should a sampling and change evidence brief include?
It should include the design statement, the replacement rules written before field, a replacement log under pseudonymous IDs, quality removal counts, the source mix, weighting, a rewritten achieved-coverage claim, withdrawal and deletion records, a minimization review and a reuse check. Each row names who signed it.
Do I need to report replacements in a quota sample?
AAPOR's substitution rule is written for sampled cases, and its Code asks researchers to describe quota use and discuss any replacement. For quota samples there is no selection probability to protect, but the coverage claim still depends on who filled the cells, so reporting the number and nature of replacements is the defensible practice.
Does removing speeders change a study's coverage?
It can. If removed cases are concentrated in particular cells and the backfills come from different sources or later in field, the achieved sample differs from the planned one. Report the checks used and the counts per cell so readers can judge.
How do I keep a replacement log privacy-safe?
Use study-scoped pseudonymous case IDs rather than panel member numbers or emails, record dates at day level, and use fixed reason codes instead of free text. Keep only the fields the brief needs and record why each one is kept.
What if a respondent asks to be deleted after the data was delivered?
Remove the case, reissue the affected files, and record the request reference, the cases removed and the new file versions in the brief. The delivered counts and any coverage statement should be updated to match.
Sources
- AAPOR's Standard Definitions (10th edition) state that any substitution of sampled cases, replacing an originally sampled unit with another, must be reported. Source
- The 2016 edition of AAPOR's Standard Definitions requires substitution rules to be documented, the number and nature of substitutions to be reported, and all replaced cases to be accounted for in final disposition codes. Source
- AAPOR's Code asks researchers to describe the sample design and any use of quotas, and says any data imputation, exclusions or replacement will also be discussed. Source
- AAPOR's 2013 task force report on non-probability sampling recounts the 1948 quota-sampling polling failure and interviewer selection of more educated, better-off people, and notes nonprobability panel samples are usually quota samples. Source
- AAPOR's report on evaluating survey quality lists whether respondents such as speeders or satisficers were removed from the final dataset as a question to answer. Source
- ESOMAR's questions for buyers of online samples ask how multiple sample sources are blended, how duplicates across sources are handled, and whether clients are notified in advance of third-party sources. Source
- ISO 20252 sets vocabulary and service requirements for market, opinion and social research; ISO lists the 2019 edition as withdrawn with ISO 20252:2026 available. Source
- The ICO explains data minimization as personal data that is adequate, relevant and limited to what is necessary, and says organizations must not collect or keep data on the off-chance it might be useful. Source
- ESOMAR publishes 37 Questions to Help Buyers of Online Samples, a framework for evaluating online sample providers. Source
See if your company qualifies
A short company assessment. No data uploads are needed.