Skip to content

AI data market

Dataset sample representativeness audit: is it the supplier's best work?

By SourceX Editorial · Updated

9 sections

Short answer

To detect a supplier sample that overrepresents its best records, have the supplier commit a hashed list of every eligible record, draw your own seeded random sample from it, and compare the two samples on quality indicators such as field fill, length, exception share, rework and label agreement. A sample whose category mix matches but whose quality is consistently higher was selected, not drawn.

Checklist

Dataset sample

This page covers dataset sample representativeness audit: is it the supplier's best work.Illustrative

Key takeaways

  • Mix-and-date checks catch a skewed sample but not a curated one; a curator can match every stratum while picking its cleanest records.
  • Ask for a sampling frame of every eligible record identifier and its hash before you send a seed, then draw your own sample from it.
  • Compare supplier and buyer samples on quality indicators with two-sample tests, within strata first, and look for a consistent tilt.
  • Adopt the buyer draw's quality profile, not the showcase profile, as the reference for delivery acceptance.
  • Curated samples are fine for checking format; the audit applies when a sample is used to judge quality or set terms.

How do you detect a supplier sample that overrepresents its best records?

Compare the sample the supplier chose with a second sample you chose, drawn at random from a record list the supplier committed to before knowing which records you would pick. If the supplier's sample scores better on quality indicators that the full population can also be measured on (field fill, length, exception share, rework, outcome mix), the first sample was selected, not drawn.

That buyer-controlled redraw is the core of the audit on this page. Ordinary representativeness checks compare a sample's category mix and date spread against population statistics, and they matter, but a careful curator can pass them by picking the cleanest records inside every stratum. The tests below are aimed at that specific failure: a sample whose shape matches the population while its quality does not.

Why a showcase sample can pass a mix-and-date comparison

A hand-picked sample can match the population on every count you check and still be the best slice of it. If the population is 40 percent refunds and 60 percent exchanges (illustrative shares), a curator can deliver exactly that ratio while choosing refunds with complete notes, short threads and a clear outcome. Stratified comparisons confirm the mix; they say nothing about which records inside each stratum were taken.

Selection on quality is rarely announced. Someone exports a folder of examples prepared for a sales deck, filters out tickets with missing fields because they look broken, or keeps only threads that a senior agent handled. Each choice is reasonable for a demo and misleading for a purchase decision, because a model trained or evaluated on the delivery will meet the records that were left out.

The audit therefore looks for two kinds of evidence: how the supplier's sample was produced, and whether its quality indicators sit above the same indicators measured on records nobody chose by hand.

Commit the record list before anyone draws from it

The redraw is only meaningful if the supplier cannot change the population after seeing which records you want. Ask the supplier to export a sampling frame, a list of every eligible record identifier with its stratum label and creation date, and to send you a cryptographic hash of that file before you send your random seed. NIST's Secure Hash Standard, FIPS 180-4, specifies hash algorithms such as SHA-256 whose message digests can be used to detect whether a message has changed since the digest was generated.

Then generate the seed yourself and draw the identifiers. Python's random module documentation says that reusing a seed reproduces the same sequence, which makes the draw auditable, and describes random.sample as returning unique elements for sampling without replacement. The same documentation warns that the module's generators should not be used for security purposes and points to the secrets module, so create the seed itself with a secure source and record it in the audit file.

This mirrors a principle from financial audit. The PCAOB's audit sampling standard, AS 2315, says sample items should be selected so that the sample can be expected to be representative of the population, and that every item in the population should have an opportunity to be selected. A supplier's curated folder fails that test by construction; a seeded draw from a committed frame passes it.

Keep the frame file lean so it can be shared before any rights review is finished. It needs only an opaque record identifier, the stratum labels you plan to compare (year, source system, record type), the creation date and an eligibility flag, with no customer names, message bodies or other content. Ask for the frame row count in the same message as the hash, and confirm both again when the drawn records arrive, so a quietly trimmed frame shows up as a mismatch.

The sample selection and representativeness audit

Run the nine checks below on two samples side by side: the supplier's sample and the buyer-drawn sample of similar size from the committed frame. Each row names the test, how to run it, the result that should worry you and the evidence to file. The audit is a working template; adjust the indicators to the record type, but keep the structure.

Order matters. Settle the selection account and the frame commitment first, because every later check depends on them; a supplier that cannot produce a frame has answered the main question already. Run the indicator comparisons next, and save the blind re-labeling for last, since it costs reviewer time and is only worth it when the cheaper checks leave the question open.

The sample selection and representativeness audit
CheckHow to run itRed flagEvidence to keep
Selection accountAsk for the query, filters, date range, seed and any manual removals behind the supplier sampleAnswer is a description, not a query; manual removals exist but are not listedWritten answer and the query text
Frame commitmentSupplier hashes the full eligible-ID list; buyer sends the seed only after receiving the hashFrame count changes between hash and draw, or the hash cannot be reproducedHash, frame row count, seed, draw script
Frame coverageCompare frame row counts by year and source system with the supplier's own volume claimsFrame is smaller than the claimed volume with no exclusion reasonCount table with the claim beside it
Identifier gapsSort sample IDs and frame IDs; compare how the sample IDs are spread across the frame's ID rangeSample clusters in a narrow ID or date band the frame does notRange plot or decile table
Fill and lengthCompare non-empty field counts per record and text length between the two samplesSupplier sample is fuller and longer in most strataPer-stratum medians and test output
Exceptions and failuresCompare the share of escalated, reopened, cancelled or disputed recordsSupplier sample has fewer hard cases than the buyer drawContingency table and test output
Rework signalsCompare edit counts, reassignment counts or time to close where the system records themSupplier sample shows cleaner workflow historyDistribution comparison
Outcome labelsCompare label mix and, on a small blind subset, re-label and measure agreementLabels in the supplier sample agree with review more oftenAgreement table per sample
ReproducibilityRerun the buyer draw from the stored seed and frameDraw cannot be reproduced exactlySecond run output

Which statistical tests separate a chosen sample from a drawn one?

Use two-sample tests on each quality indicator, one indicator at a time, and read them as signals that need an explanation rather than as verdicts. For continuous indicators such as text length, number of populated fields or days to close, a two-sample Kolmogorov-Smirnov test compares whole distributions; SciPy's ks_2samp documentation describes it as comparing the underlying continuous distributions of two independent samples, whose sizes can differ.

For categorical indicators such as outcome, escalation flag or reopened status, build a table with one row per sample and one column per category and run a chi-square test of independence. SciPy's chi2_contingency documentation describes this test and computes expected frequencies from the table's marginal sums under the assumption of independence. A small p-value says the two samples differ in that category mix; it does not say which one is right.

Plan the comparison before the data arrives. Name the indicators, the direction that would suggest curation (fuller, longer, fewer exceptions) and how you will handle running many tests at once, because with a dozen indicators one will look unusual by chance. A consistent tilt in the same direction across several indicators is stronger evidence than any single small p-value.

Identifier and date patterns need no statistics at all. Sort the supplier sample by record identifier and by creation date and lay both over the frame. A random draw spreads across the whole range; a curated folder tends to bunch in the weeks after a template change, a system migration or the arrival of one strong team. A bunch is not proof, but it tells you where to look in the selection account.

  • Compare within strata first, then overall, so a mix difference does not masquerade as a quality difference.
  • Report effect sizes such as the difference in medians or shares, not only test statistics.
  • Keep both samples large enough that each stratum you care about has records in it; otherwise merge strata and say so.

Worked example: insurance claim notes offered for a summarization model (illustrative)

This example is invented to show the audit's mechanics; the counts and percentages are illustrative and describe no real supplier. A claims administrator offers adjuster notes for training a claim-summary model and sends a sample of 400 claims. The buyer asks for the frame, receives a hash of a list of eligible claim identifiers, sends a seed and draws 400 claims of its own.

Category mix lines up: both samples have roughly the same split across auto, property and liability claims and across years. The quality indicators do not. In the supplier sample, a large majority of claims have every note field populated, against a clearly lower share in the buyer draw; the share of reopened or litigated claims in the supplier sample is about half the share in the buyer draw; and note length is longer at every percentile. The Kolmogorov-Smirnov and chi-square tests point the same way.

The supplier explains that the sample came from a team that piloted structured note templates. That is a legitimate reason, and it changes the decision rather than ending it: the buyer now knows that the bulk of the archive looks like its own draw, prices the cleaning effort on that basis and writes the buyer-draw profile, not the showcase profile, into the acceptance criteria.

The exception in this case is worth noting. The litigated claims were absent from the supplier sample partly because counsel had placed some under a hold. That is a rights issue, not a quality issue, and it is handled in the licensing review rather than in the statistics.

What must the pilot establish before the larger purchase?

The pilot should prove that the delivery will look like the buyer draw, not like the showcase, and it should put that expectation in writing. Before committing to volume, settle the points below with the supplier and carry them into the agreement and acceptance terms.

Keep the pilot small and quick on purpose. Its job is to confirm that the reference profile holds on a fresh seeded draw from the delivery frame, and that the cleaning or labeling effort you priced from the buyer draw is the effort you actually need. If the pilot draw drifts toward the showcase profile or away from it, find out why before the volume order, not after.

  • The frame definition: which records are eligible, which are excluded, and why, with counts.
  • The quality profile of the buyer draw for each indicator, adopted as the reference for the delivery.
  • Tolerances for each indicator, for example a delivered fill rate within a stated number of points of the reference profile, set before the delivery is cut.
  • The right to repeat the seeded draw on the delivery itself and reject or cure batches that miss tolerance.
  • An exceptions log for records withheld for privacy, legal hold or contractual reasons, so missing hard cases are explained rather than silently absent.
  • Who reviews what: procurement owns the profile, a research reviewer owns the test plan, and security and legal review the frame export and any withheld records.

When a curated sample is the right thing to ask for

Curated samples are useful when the question is format, not quality. A small hand-picked set is the fastest way to check field names, file structure, de-identification approach and how a model ingests the records, and nobody should be blamed for sending one. The audit applies when a sample is used to judge quality, size the cleaning effort or set acceptance terms.

Some selection is also legitimate and should be documented rather than removed. Rare strata may be deliberately oversampled so they can be inspected; records may be held back for privacy or legal reasons. Datasheets for Datasets asks dataset creators to state whether a dataset is a sample of a larger set, whether it is representative of that set and how representativeness was validated, and, if not, why not. A supplier who answers those questions in writing has already done half of this audit.

Stop and bring in specialists when the frame itself cannot be produced, because without a frame there is no population to compare against; when withheld records are concentrated in one claim type, customer segment or period; or when the quality gap is large enough that the price, cleaning budget or task fit would change.

Bringing the audit to a sourcing conversation

The audit works best when it is agreed before the first sample leaves the supplier, so the frame, seed handling and indicators are part of the request rather than a surprise.

The frame, seed handling and indicator list can travel with the request: SourceX sources datasets from U.S. businesses when a buyer asks rather than keeping stock, and its request form already collects scope, volume, language, format, timing and licensing needs. On the supply side, the company approves scope and terms and signs before any data is delivered.

Frequently asked questions

What is a sampling frame for a dataset purchase?

A sampling frame is the full list of records eligible for delivery, usually record identifiers with their stratum labels and creation dates. Drawing samples from a committed frame lets a buyer check that every eligible record had a chance of being selected. Without a frame, a sample can only be compared with the supplier's own summary statistics.

Why should the buyer, not the supplier, choose the random seed?

If the supplier chooses the seed after seeing the frame, it can rerun the draw until a favorable sample appears. Having the supplier commit to the frame with a hash first, then having the buyer supply the seed, removes that option. Storing the seed and script also makes the draw reproducible during acceptance.

How large should the buyer-drawn sample be?

Large enough that every stratum you plan to compare contains records, and similar in size to the supplier's sample so the tests compare like with like. If some strata end up nearly empty, merge them and state that in the audit file. A statistician can size the draw for the smallest quality difference that would change your decision.

Is a curated sample a sign of bad faith?

Usually not. Suppliers often send their cleanest records to show format and structure. It becomes a problem only when the curated sample is used to judge quality or set acceptance terms, so ask for a seeded draw before those decisions.

What if the supplier cannot export a full record list?

Ask for the narrowest frame that can be produced, such as one system or one year, and limit the purchase decision to that scope. If no frame can be produced at all, treat the sample as a format preview only and move quality checks into a paid pilot with acceptance rights.

Sources

  • PCAOB AS 2315 states that sample items should be selected so the sample can be expected to be representative of the population, with every item having an opportunity to be selected. Source
  • FIPS 180-4 specifies secure hash algorithms, including SHA-256, whose message digests can detect whether a message has changed. Source
  • Python's random module documentation covers seed reproducibility, random.sample for sampling without replacement, and the warning not to use it for security purposes. Source
  • SciPy's ks_2samp performs a two-sample Kolmogorov-Smirnov test comparing the continuous distributions of two independent samples of possibly different sizes. Source
  • SciPy's chi2_contingency runs a chi-square test of independence on a contingency table, computing expected frequencies from marginal sums. Source
  • Datasheets for Datasets asks whether a dataset is a sample of a larger set, whether it is representative and how representativeness was validated. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies