How to request a training data sample from a supplier

Match the sample stage to the decision it supports

A sample request starts from the decision the sample informs, because a schema preview, a look sample, an evaluation sample and a trial slice prove different things. A purchase can pass through several of these stages between RFI and contract in the AI training data procurement lifecycle.

Conventions differ by market. The FISD Alternative Data Council's illustrative due diligence questionnaire, written for data sold to investment managers, asks providers for a data dictionary and a sample of fewer than 100 rows from data more than three months old [1], which is a look sample: enough to read the schema, too small to measure anything. One licensor's documented process includes a sample stage in which the buyer evaluates the sample and any iterations happen before final delivery [2].

If the holder will not release raw records, the options narrow to secure viewing, redacted records or synthetic stand-ins; see what restricted sample routes can and cannot prove.

How big a data sample should be for each test

There is no single answer to how big a data sample should be: size it from the test it feeds. A schema check needs every record type, a defect-rate check enough random records to rule out the rate you care about, and a model test enough examples to move a metric your evaluation set can resolve.

Your evaluation set limits what an ablation can show: one practitioner guide calculates that detecting a lift from 82% to 85% accuracy at 80% power and 5% significance needs about 2,400 labeled examples per model variant [8]. Measurement methods are in estimating a dataset's value before you buy.

For repeated deliveries, ANSI/ASQ Z1.4 tables give sample sizes and accept/reject numbers by lot size, inspection level and acceptable quality level (AQL) for attribute inspection [3]. The standard was written for product lots, so define how a delivery batch and a defective record map onto it in your acceptance criteria for licensed training data.

Worked example: sizing one request for three tests

Illustrative example: invented to show structure; it does not describe an available dataset.

A team plans a residual-PII check, a supervised fine-tuning (SFT) ablation and a retrieval test on a support-ticket dataset. The supplier says fewer than 1% of records retain an identifier.

1. PII check. If the true rate were 1%, the chance that n random records contain no identifier is 0.99^n: about 37% at n = 100, about 5% at n = 300. A clean 300-record review therefore rejects "1% or worse" with roughly 95% confidence. Bounds for other rates are in sample sizes for estimating a dataset's error rate.
2. SFT ablation. Ask for 2,000 records by simple random draw and review 300 of them for step 1, so one draw serves both tests.
3. Tail coverage. Add 400 records oversampled from three rare strata (escalations, refunds, non-English tickets), each tagged with its sampling weight.
4. Retrieval test. Request every ticket and linked knowledge-base article for one product line and quarter, because a retrieval test needs the documents that compete with the right answer.

Specify how records are selected, not only how many

The selection method decides whether results transfer to the full dataset, so the request names it: a simple random draw with a recorded seed, or a stratified draw across strata you choose. A supplier-curated "best of" set answers none of your questions. Ask for the selection query or script, the seed and the population count behind each stratum.

Pick strata that drive model behavior in your use case: source system, year or quarter, record type, language, customer segment, outcome (resolved, escalated, refunded) and length bands for free text. Then choose an allocation. Research on data representativity separates two goals: a sample that mirrors the population, which supports average-case estimates, and one that covers the input space, which helps robustness to distribution shift and accuracy on smaller subgroups [9]. A proportional random core plus a weighted oversample of the tail can serve both goals.

Ask for full-population statistics in the same request: counts per stratum, field fill rates, date histograms, text-length percentiles, label distributions and records removed by each filter. The comparison method is in checking whether a vendor's sample is representative, tracing sampled records to consent evidence is in testing a supplier's provenance claims on a sample, and the supplier's view is in the supplier-side guide to selecting a representative sample.

Require the sample to be prepared exactly like the full delivery

A sample predicts the full dataset only if it passes through the same preparation pipeline, at the same version, as the records you will license. A hand-cleaned or separately exported sample hides the defects you are paying to find. Spell out parity on five points:

• De-identification. The same method, entity types, replacement scheme (consistent surrogate values or generic masks) and tool version. Automated detection misses things: the open-source Presidio project cautions that because it uses trained models there is no guarantee it will find all sensitive information [10]. For protected health information covered by HIPAA, name the method: Safe Harbor removal of 18 identifier types, or Expert Determination [11].
• Filters and deduplication. The same language, date and quality filters, applied before sampling. Near-duplicates are common in web-scale text corpora; one study found a single sentence repeated more than 60,000 times in the C4 corpus [12]. A sample drawn before deduplication misstates the duplicate rate you will receive.
• Schema and IDs. The schema version, field names and stable record IDs that will appear in the full delivery, so every sample finding maps to a delivered record.
• File format and writer settings. The same container, compression and writer. Parquet implementations do not all support the same features of the format [13], so a sample written by one library can read cleanly while the full delivery fails in yours.
• Manifest. File list, record counts and checksums in the form you expect at delivery. An illustrative package summary and field guide shows the level of detail to ask for.

Documents to request with the sample

Ask for the documents that let you interpret the records: a data dictionary, a selection log, a preparation log, population statistics and a datasheet. A supplier who knows its data can produce them, so a gap is itself a finding.

• Data dictionary: field names, types, units, allowed values, null semantics and code lists. The FISD template asks for one alongside its sample [1].
• Selection log: query, seed, strata, weights and population counts.
• Preparation log: de-identification method and version, filters, deduplication method and threshold, records removed per step.
• Population statistics for the full candidate dataset, as listed above.
• Datasheet answers on motivation, composition, collection process and recommended uses, following Datasheets for Datasets [14].
• Machine-readable metadata where available, such as Croissant, a schema.org-based JSON-LD vocabulary covering dataset, file and record structure [15]. As of October 2026, NeurIPS requires responsible-AI metadata based on Croissant RAI fields for its Evaluations and Datasets Track [16], a sign of where documentation norms are heading.
• For labeled data: annotation guidelines, the multiply-annotated subset and agreement statistics.
• Known issues: drifting fields, date gaps, mid-period system changes.

The wider pre-signature evidence pack, including profiling reports and provenance summaries, is covered in evidence to request from a data vendor before you sign. If you source through SourceX, you can put this list straight into a data request on the SourceX buyer page. SourceX looks for US businesses that hold the data you describe, and prepares diligence materials on source, rights, preparation and allowed use per dataset for your review.

Sample terms that permit the tests you planned

Before the sample ships, confirm in writing that the evaluation terms allow every test in your request, including any fine-tuning run, and state what happens afterward to the records, checkpoints and results; evaluation grants are often narrower than buyers assume.

NVIDIA's sample data license for evaluation (2026.01.19 version) grants a limited, non-exclusive, revocable, non-transferable, non-sublicensable right solely to evaluate and test NVIDIA technologies, and bars other uses and redistribution [17]. A grant worded that way would not cover training your own model on the sample. One data vendor's template combines a trial data license and a mutual NDA in one document [18]. Check five points:

1. Permitted tests. Name each test, plus the internal teams and cloud environments involved.
2. Results. You keep metrics, error analyses and evaluation reports, whether or not you buy.
3. Retention. A deletion date for the sample and any checkpoints trained on it, with written certification.
4. Overlap with the full delivery. Sample record IDs flagged in the full delivery, so records you used for evaluation stay out of training; see contamination checks for licensed evaluation data.
5. Re-identification. As of October 2026, under the CCPA definition in California Civil Code section 1798.140(m), information counts as deidentified only if, among other conditions, the business contractually obligates any recipients to comply with all of the definition's provisions, which include not attempting to re-identify it [19]. A supplier relying on that status therefore has reason to include the clause in sample terms.

Negotiating these terms is covered in evaluation licenses and NDAs for dataset samples, and fees and conversion credits for larger trials in paid data pilot terms. Suppliers' own expectations are described in the supplier-side explanation of sharing samples.

Illustrative dataset sample request

Putting size, selection, preparation, documents and terms in one written request lets you compare suppliers' answers line by line.

Illustrative example: invented to show structure; it does not describe an available dataset. Not legal advice; adapt with counsel.

sample_request:
  candidate_dataset: "B2B software support tickets with agent replies, 2021-2025"
  planned_tests: [pii_residue_review, sft_ablation, retrieval_test]
  selection:
    core:
      records: 2000
      method: simple_random
      seed: 20261009
    tail_oversample:
      records: 400
      strata: [escalated_to_engineering, refund_issued, non_english]
      report_sampling_weights: true
    retrieval_slice: "all tickets and linked KB articles, one product line, one quarter"
  preparation_parity:
    pipeline_version: same_as_full_delivery
    deidentification: "same method, entity list and surrogate scheme; method and version named"
    filters_and_dedup: "applied before sampling; thresholds stated"
    format: "JSONL, UTF-8, one ticket per line; same schema version and record IDs"
  documents:
    - data_dictionary
    - selection_log          # query, seed, strata, population counts
    - preparation_log        # steps, versions, records removed per step
    - population_statistics  # per stratum: counts, fill rates, date histogram, length percentiles
    - datasheet
    - manifest_with_checksums
  terms:
    permitted_tests: [profiling, manual_review, fine_tuning_ablation, retrieval_test]
    buyer_keeps_results: true
    delete_sample_and_checkpoints: "within 60 days of decision, certified"
    flag_sample_ids_in_full_delivery: true
    no_reidentification: true

Red flags when the sample arrives

Treat a sample as a preview of how the supplier will behave at delivery, so read these signs as findings to resolve before you sign:

• No selection log, or "selected for quality" given as the method.
• Every record from one year, customer or system when the dataset is described as multi-year or multi-source.
• A different format, schema version or ID scheme from the one quoted for delivery.
• Zero residual identifiers and no preparation log, which suggests hand-cleaning.
• Population statistics that cannot be reconciled with the sample's stratum counts.
• Evaluation terms that arrive after the data, or that forbid the tests you named.

Rates measured on a sample that passes become the baseline for delivery thresholds. Modality-specific tests are covered in evaluating a code dataset sample, evaluating an agent data sample and pilot-testing a content source for retrieval lift.

Describe the dataset and sample you need

If you need operational records from US businesses, describe the dataset and the sample specification you will test against on the SourceX buyer page. SourceX looks for US companies that hold the data you describe, checks the data and the supplier's licensing permissions, and manages the license and delivery; a request does not guarantee a matching dataset. Describe the data and sample you need.