Skip to content

Privacy and preparation

Market research sample replacement: how swaps change coverage claims

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A study should account for replaced respondents by logging every replacement before the data leaves the agency: the rule that allowed it, the stage, the reason, the quota cell, the source of the substitute and the date, under study-scoped pseudonymous IDs. Report the number and nature of replacements, and restate the coverage claim as achieved rather than as designed whenever replacements cluster in particular cells or sources.

License terms

Market research sample

This page covers market research sample replacement: how swaps change coverage claims.Illustrative

Key takeaways

  • Every replacement is an event with a rule, a stage, a reason, a cell, a source and a date.
  • Matching a quota cell does not show that a replacement was neutral; clustered swaps move the claim.
  • AAPOR requires substitutions to be reported, and its Code asks for quotas and replacements to be described.
  • The replacement log is respondent data, so it uses pseudonymous IDs, day-level dates and fixed reason codes.
  • Rewrite the claim as achieved, not as designed, before a study's records are reused or licensed.

How should a study account for replaced respondents in its coverage claims?

It should treat every replacement as an event with evidence, and let the coverage claim follow the evidence. A study designed as a national quota sample of 1,000 adults and delivered as 1,000 completes can still describe a different population if a large share of completes were swapped in late, from a different panel, into a few cells. The sentence in the report should describe what was achieved.

Survey research standards already require this for probability samples. AAPOR's Standard Definitions say that any substitution of sampled cases, replacing an originally sampled unit with another, must be reported, and the earlier edition spells out three parts: document the substitution rules, report the number and nature of substitutions, and account for every replaced case in the final disposition codes. The same discipline works for quota and panel studies, where the claim rests on how the sample was assembled.

This matters more when a study's records are reused. When an agency licenses respondent-level data for AI training or evaluation, the buyer inherits the coverage claim. A claim that no longer matches the records travels with the dataset and misleads every model built on it.

Where replacement happens in a market research study

Replacement is not one event. It happens at several stages, for different reasons, and each changes the sample in its own way. Agencies usually keep the evidence for some of these in the survey platform, some in the panel provider's reports and some only in a project manager's email.

Start by listing which of these stages a study actually went through, then find where each one left a trace. Survey platforms usually hold disposition and quota status data for each case, panel providers send completion and reconciliation reports, and the data processing team keeps the cleaning script that applied the quality checks. Pull those into one place before anyone writes the methods note, because they are hard to reassemble a year later.

Where replacement happens in a market research study
StageTypical triggerWhat it can changeWhat to log
FieldworkInvitee does not respond; cell filled from fresh invitesWho answers: quicker, more active panelistsInvite wave, cell, source, date filled
Quality removalSpeeding, straight-lining, failed attention checksCells where removed cases were concentratedCheck failed, count by cell, backfill source
Quota managementCell closes early or overfills; late cells backfilledBalance between early and late respondersCell close dates and backfill counts
Source blendingSecond panel or third-party supplier added mid-fieldMix of recruitment methods within cellsSupplier per case, date added, client notice
Tracker attritionWave respondents drop out; refreshment sample addedComparability between wavesRetained versus fresh cases per wave
Qualitative recruitingNo-show replaced from the screener backlogFit against screener criteriaOriginal and replacement screener answers
After fieldworkWithdrawal or deletion requestDelivered counts and cell totalsRequest date, cases removed, files updated

Why the claim moves even when every quota still adds up

Quotas control the traits you set them on, usually age, gender, region and perhaps income. A replacement that fits the cell can differ on everything else: how fast the person answers, how many surveys they take, which panel recruited them. When replacements cluster, those unmeasured differences cluster too.

The history of quota sampling is a reminder. AAPOR's 2013 task force report on non-probability sampling recalls the 1948 U.S. election, where the major polls relied on quota sampling and called the winner wrong, and the review that followed found that interviewers had tended to pick somewhat more educated and better-off people within their quotas. The report notes that today's nonprobability panel samples are usually quota samples. The lesson for a replacement log is simple: matching the cell does not show that the swap was neutral.

In practice, three patterns deserve a closer look. Backfills that arrive in the last days of field come from people who were slower to respond or were invited later. Backfills from a different supplier come from people recruited by different methods and incentives. Backfills concentrated in one or two cells mean those cells now describe a different mix of people from the rest of the sample. None of these is wrong in itself; each one has to be visible in the claim.

Which disclosure rules already cover replacements?

Several published standards ask for pieces of this evidence, so the brief below mostly gathers what a careful agency should already hold. AAPOR's Code of Professional Ethics and Practices asks researchers to describe the sample design and any use of quotas, and says any data imputation, exclusions or replacement will also be discussed. AAPOR's report on evaluating survey quality lists whether respondents such as speeders or satisficers were removed from the final dataset among the questions a study should answer.

For online samples, ESOMAR's questions for buyers of online samples ask how multiple sample sources are blended, how duplicate respondents across sources are handled, and whether clients are told in advance when a third-party source is used. ESOMAR now publishes a 37-question edition of that guide. ISO 20252 sets vocabulary and service requirements for market, opinion and social research; ISO lists the 2019 edition as withdrawn and replaced by ISO 20252:2026, so check the current text for the clauses your certification body audits.

The privacy-reviewed sampling and change evidence brief

Prepare one brief per study, or per wave for a tracker, before any respondent-level data is licensed or shared. Each row names its evidence and its privacy treatment, because a replacement log is itself respondent data. The ICO describes data minimization as holding personal data that is adequate, relevant and limited to what is necessary, and warns against keeping data on the off-chance it might be useful; the log should carry reason codes and pseudonymous IDs, not names, emails or panel member numbers.

At a glance

The privacy-reviewed sampling and change evidence brief

The privacy-reviewed sampling and change evidence brief
Brief fieldWhat it containsPrivacy treatmentSigned by
Design statementTarget population, frame, mode, sources and planned quotasNo personal dataProject lead
Replacement rulesRules for backfill, removal and blending, dated before fieldNo personal dataProject lead
Replacement logCase ID, stage, reason code, cell, source, dateStudy-scoped pseudonymous IDs; day-level datesField manager
Quality removalsChecks used, thresholds, counts removed per cellCounts only in the brief; case list kept internalData processing lead
Source mixShare of completes per supplier, per cell, with client noticeSupplier names are business data, not respondent dataSample manager
WeightingVariables, targets and their sources, or a statement of no weightsAggregate targets onlyAnalyst
Achieved coverageRewritten claim describing the delivered sampleNo personal dataResearch director
Withdrawals and deletionsRequests received, cases removed, versions updatedRequest references, not identitiesPrivacy lead
Minimization reviewFields dropped from the log and why each kept field is neededRecord of the review itselfPrivacy lead
Reuse checkNotice and contract terms that cover the intended reuseClause references onlyRights reviewer

Worked example (illustrative): one wave of a consumer tracker

The counts below are invented to show the method; they describe no real study. A tracker wave targets 1,000 completes of U.S. adults with quotas on age, gender and region from one main panel. During field, the agency removes 84 completes for failing quality checks and backfills them. The two youngest male cells run short, so the sample manager adds a second panel for the last three days, which supplies 61 of the backfills. After delivery, 6 respondents ask to be deleted.

  • Designed claim: a nationally representative sample of 1,000 U.S. adults.
  • Achieved claim (illustrative): a quota sample of U.S. adults matched on age, gender and region; 994 completes after 6 deletions; 84 quality replacements, 61 of them in two younger male cells from a second panel added late in field.
  • Brief entries: replacement log under wave-scoped IDs, check thresholds, the supplier change with the date the client was told, and a deletion record for each affected file version.
Worked example (illustrative): one wave of a consumer tracker
Event (illustrative)CountWhere it landedEffect on the claim
Quality removals backfilled84Spread across cells, heavier among younger respondentsReport removals and checks used
Backfills from a second panel61Two youngest male cells, final three daysName blending; claim no longer single-source
Cells closed early5Older respondent cellsOlder cells are early responders only
Post-delivery deletions6Mixed cellsDelivered total and cell counts change

The exception: removals that never belonged in the sample

Some removals do not move the coverage claim. Duplicate completes, bots and respondents who were never eligible were not valid members of the sample, so removing them restores the design rather than changing it. Log them anyway with a reason code, because a buyer cannot tell a fraud sweep from a quota rescue unless the log says which is which.

Qualitative recruiting is a second case. A replaced interviewee in a 20-person study does not change a statistical coverage claim, because there is none; what matters is whether the replacement met the same screener criteria. Keep both screener records, pseudonymized, so the fit can be checked later.

Which service condition or record shows the obligation was met?

Two different obligations are in play, and they are met by different evidence. The client's specification, usually in the proposal or statement of work, sets counts, quotas, incidence, quality checks and sometimes approved sources; the field disposition file and the replacement log show whether those were met. The coverage claim in the report is a separate statement about the population, and only the achieved-coverage row in the brief can support it.

Keep the link between them explicit. If the client approved a second panel by email, attach that approval to the source mix row. If a deletion request arrived after delivery, record which delivered files were reissued. A dataset reviewer should be able to trace each change from the claim back to a dated record without reading anyone's inbox.

When the specification and the claim disagree, the specification usually decides whether the agency delivered what was bought, and the claim must be narrowed to match. An agency that met every quota with late backfills from a second source has met its service obligation and still owes the reader an accurate description of the sample.

When to hold a study out of a licensed dataset

Hold a study back, or license it only with a narrowed description, when its sample history cannot be reconstructed or when the log itself raises privacy questions. A qualified survey methodologist should read the brief for any study whose coverage claim will be repeated to a buyer, and a privacy or rights reviewer should confirm that respondent notices and client contracts allow the reuse at all; de-identification alone does not create that permission.

  • Replacement rules were written after fieldwork, or not at all.
  • Sources were blended without a record of which cases came from where.
  • Reason codes are free text that describes individuals.
  • Deletion or withdrawal requests cannot be traced to the delivered files.
  • The client contract or respondent notice does not cover the intended reuse.
  • The achieved sample differs from the claim and nobody will sign the rewrite.

What a licensing review asks about sample history

A licensing review asks for the brief, not the respondents. SourceX licenses business data directly from agencies and other companies, and an agency's first step is a short company assessment with no data upload. The agency approves the scope and terms and signs before any respondent-level file is delivered, so studies with incomplete sample histories can be left out of scope from the start.

Frequently asked questions

What should a sampling and change evidence brief include?

It should include the design statement, the replacement rules written before field, a replacement log under pseudonymous IDs, quality removal counts, the source mix, weighting, a rewritten achieved-coverage claim, withdrawal and deletion records, a minimization review and a reuse check. Each row names who signed it.

Do I need to report replacements in a quota sample?

AAPOR's substitution rule is written for sampled cases, and its Code asks researchers to describe quota use and discuss any replacement. For quota samples there is no selection probability to protect, but the coverage claim still depends on who filled the cells, so reporting the number and nature of replacements is the defensible practice.

Does removing speeders change a study's coverage?

It can. If removed cases are concentrated in particular cells and the backfills come from different sources or later in field, the achieved sample differs from the planned one. Report the checks used and the counts per cell so readers can judge.

How do I keep a replacement log privacy-safe?

Use study-scoped pseudonymous case IDs rather than panel member numbers or emails, record dates at day level, and use fixed reason codes instead of free text. Keep only the fields the brief needs and record why each one is kept.

What if a respondent asks to be deleted after the data was delivered?

Remove the case, reissue the affected files, and record the request reference, the cases removed and the new file versions in the brief. The delivered counts and any coverage statement should be updated to match.

Sources

  • AAPOR's Standard Definitions (10th edition) state that any substitution of sampled cases, replacing an originally sampled unit with another, must be reported. Source
  • The 2016 edition of AAPOR's Standard Definitions requires substitution rules to be documented, the number and nature of substitutions to be reported, and all replaced cases to be accounted for in final disposition codes. Source
  • AAPOR's Code asks researchers to describe the sample design and any use of quotas, and says any data imputation, exclusions or replacement will also be discussed. Source
  • AAPOR's 2013 task force report on non-probability sampling recounts the 1948 quota-sampling polling failure and interviewer selection of more educated, better-off people, and notes nonprobability panel samples are usually quota samples. Source
  • AAPOR's report on evaluating survey quality lists whether respondents such as speeders or satisficers were removed from the final dataset as a question to answer. Source
  • ESOMAR's questions for buyers of online samples ask how multiple sample sources are blended, how duplicates across sources are handled, and whether clients are notified in advance of third-party sources. Source
  • ISO 20252 sets vocabulary and service requirements for market, opinion and social research; ISO lists the 2019 edition as withdrawn with ISO 20252:2026 available. Source
  • The ICO explains data minimization as personal data that is adequate, relevant and limited to what is necessary, and says organizations must not collect or keep data on the off-chance it might be useful. Source
  • ESOMAR publishes 37 Questions to Help Buyers of Online Samples, a framework for evaluating online sample providers. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies