Skip to content

AI data market

How to run a dataset provenance maturity survey: the study protocol

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A study can assess whether companies can prove where their training data came from by asking each sampled company to show documents for one randomly chosen dataset, then scoring verified evidence for origin, terms, rights, processing, personal data and delivery. Report levels with intervals, response rates and the self-rating gap. This page is the protocol only; it reports no results.

Provenance trail

Dataset provenance maturity

This page covers how to run a dataset provenance maturity survey: the study protocol.Illustrative

Key takeaways

  • Measure evidence a reviewer inspects, not self-rated maturity.
  • Sample one dataset per company at random, with no substitutions, so participants cannot show only their best-documented data.
  • Score eight domains on four levels and let the weakest core domain set the overall level.
  • Record a disposition for every invited company and publish bounds that assume low maturity for those who decline.
  • Pre-register questions, frame, instrument, metrics and the commitment to publish; no results are reported here.

How could a study assess whether companies can prove where their training data came from?

Ask each company to show documents for one randomly chosen dataset, not to rate itself. A reviewer then scores whether evidence exists, and holds up on a sample of records, for origin, collection terms, rights, transformations, personal data handling and delivery. The study reports the share of datasets at each level with intervals, response rates and the gap between self-rating and verified evidence.

This page is a research protocol, not a report of findings. No survey has been run under it, and no results, percentages or rankings appear here. What follows is the design: the question, population, sampling, instrument, variables, comparison, metrics, uncertainty, limits and a pre-registration checklist that a research team, trade body or journalist could adopt and publish before collecting any data.

The reason to insist on evidence is simple. Self-assessed maturity surveys measure confidence, and confidence about data origins is easy to state and hard to check. A design that inspects the documents behind one dataset per company measures what a buyer, auditor or regulator would actually be shown.

What the survey counts as provenance, and what it leaves to lawyers

The survey uses the W3C definition: provenance is information about the entities, activities and people involved in producing a piece of data, which can be used to judge its quality, reliability or trustworthiness. That frames the measurement as a question about records that exist, not about whether a use is lawful.

Four things are kept apart in the instrument. Access is whether someone can technically export the data. Privacy permission is whether personal information in it may be used for a given purpose. Ownership is who holds title to the records or the intellectual property in them. Contractual licensing rights are what the company may grant to others. De-identifying a dataset does not by itself create a right to license it, so the instrument scores rights evidence separately from privacy evidence.

Several published frameworks name what provenance evidence should cover, and the instrument borrows their categories rather than inventing new ones. The Data and Trust Alliance describes its Data Provenance Standards as cross-industry metadata standards for the origin of datasets used in traditional data and AI applications. NIST's Generative AI Profile (NIST AI 600-1) suggests actions such as documenting the origin of training data, including its copyright, licensing and privacy status, and keeping audit trails of sources and transformations. The Datasheets for Datasets proposal by Gebru and colleagues sets out questions about a dataset's motivation, composition and collection process.

Prior audits show the method is feasible on public data. The Data Provenance Initiative documented sources, licenses and creators across 44 data collections spanning more than 1,800 fine-tuning datasets. That audit covered publicly released datasets; this protocol targets something different, the internal records companies hold about data they own or license.

Research questions and the conclusions this design cannot support

State the questions before recruiting anyone, and state the conclusions the design will not support with the same prominence. Both belong in the pre-registration.

At a glance

Research questions and the conclusions this design cannot support

  • Primary question: among sampled datasets held by companies in the defined population, what share has verified evidence at each maturity level in every core domain?
  • Secondary question: how far do companies' self-ratings differ from the verified level for the same dataset and domain?
  • Secondary question: which domains most often set the overall level, meaning they are the weakest link?
  • Exploratory only: do levels differ by company size band, sector or data role (supplier of records versus developer training models)?
  • Not supported: claims about companies outside the sampling frame, about all datasets a company holds, or about causes of maturity differences.
  • Not supported: any judgment that a dataset may lawfully be used or licensed. The reviewer scores the existence and consistency of documents; counsel decides legal sufficiency on specific facts.

Who is in the population, and how one dataset per company is drawn

Define the population as companies, not datasets, then draw one dataset per company at random from a list the company provides. This prevents the most common bias in provenance studies, where each participant shows its best-documented dataset.

A workable definition is companies above a stated employee threshold, in named countries, that either hold business records they have licensed or are considering licensing for AI, or train or fine-tune models on data they did not create. Write the exact eligibility screen into the protocol, including how employee counts are measured and on what date.

No complete list of such companies exists, so the sampling frame will be imperfect. Options include business registers filtered by size and sector, membership lists of industry associations, and public lists of AI developers. Record which frame is used, what it is known to miss, and stratify by size band and sector so small strata are not lost. Recruitment dates, invitation wording, the number of reminders and any non-cash incentive go in the protocol before the first contact.

Within each participating company, ask for a list of datasets that meet a stated definition (for example, a dataset prepared for or used in model training in a stated period), number them, and select one with a random number generator whose seed is logged. If the company refuses the selected dataset, record a refusal at that stage instead of accepting a substitute.

Size the sample for precision, not for a hoped-for result. For a proportion, the half-width of a 95% interval is at most about 1.96 times the square root of 0.25 divided by n. As planning arithmetic only, n of 400 completed cases gives a worst-case half-width near 4.9 percentage points; strata will be wider. Adjust for expected eligibility and response before setting invitation counts.

The evidence-based provenance maturity instrument

The instrument scores eight domains at four levels. Level 0 means no evidence; level 1 means the company asserts the fact but produces nothing; level 2 means a document exists and is produced; level 3 means the document is produced and checks out against a sample of the dataset's records. The table defines levels 1 to 3 and the evidence the reviewer inspects.

The evidence-based provenance maturity instrument
DomainLevel 1: assertedLevel 2: documentedLevel 3: verified on a record sampleEvidence the reviewer inspects
Origin systems and datesNames the source systemsSystem list with export dates and record countsSampled records trace to a named system and date rangeExport logs, system inventory, record IDs
Collection terms and noticesStates users were informedNotice or terms versions with effective datesEach sampled record falls under a produced versionPolicy versions, collection timestamps
Rights chainClaims ownership or licenseContracts, licenses or assignments producedSampled records fall within the produced grantsAgreements, IP assignments, scope clauses
Third-party contentStates none is includedScreening method written downSample screened with results loggedScreening log, exclusion list
Personal data handlingStates data is de-identifiedMethod and responsible party documentedSample checked against the stated methodMethod note, test log, exceptions
Transformations and lineageDescribes processing looselyStep list with versions and datesSample reproduced from source through the stepsPipeline config, version tags, diffs
Version and deliverySays a copy was sentManifest with file list and countsHashes or counts match the delivered copyManifests, checksums, delivery receipts
Dataset documentationInformal descriptionDatasheet or equivalent completedDatasheet statements match the sampleDatasheet, data dictionary

Scoring rules: why the weakest domain sets the level

The overall level for a dataset is the lowest level reached in any of the six core domains, not the average. A dataset with verified origin and lineage but no rights evidence cannot be licensed with confidence, and an average would hide that.

Core domains are origin, collection terms, rights chain, personal data handling, transformations and version and delivery. Third-party content and documentation are reported separately because they do not apply equally to every dataset type; write the applicability rule into the protocol.

Two reviewers score a pre-set share of cases independently, chosen at random, and the study reports agreement by domain with a chance-corrected statistic. Disagreements go to a third reviewer, and the rule for that is fixed in advance. Reviewers score from a written rubric with examples of each level and see the company's self-rating only after scoring.

The self-rating is collected first, in the same eight domains on the same four-level scale, so the comparison between claimed and verified levels uses identical categories. Report the gap as the distribution of differences, not just a mean.

Scoring rules: why the weakest domain sets the level
MetricDefinitionUnit of analysisUncertainty reported
Verified level distributionShare of datasets at overall levels 0 to 3Dataset95% interval per level, weighted and unweighted
Weakest-link domainDomain that sets the overall levelDatasetCounts with intervals; ties reported
Self-rating gapSelf-rated level minus verified level, per domainDataset and domainDistribution and interval for the median
Reviewer agreementChance-corrected agreement on double-scored casesDomainPoint estimate with interval
Completion by stageShare completing screen, self-rating and evidence reviewCompanyCounts at each stage

Response rates, uncertainty and the companies that never answer

Report every invited company in a disposition, and calculate response rates from those dispositions, because the companies least able to show evidence are the most likely to decline. The American Association for Public Opinion Research (AAPOR) Standard Definitions sort cases into completed interviews, eligible nonrespondents, cases of unknown eligibility and ineligible cases, and define a response rate as completes divided by eligible units.

AAPOR also warns that the estimate of how many unknown-eligibility cases are actually eligible must rest on the best available evidence and must not be chosen to raise the response rate. In December 2025 it published Standard Definitions for Establishment Surveys, which is the closer fit for a survey of companies, and the protocol should name which edition it follows.

Add a second, study-specific disposition layer for the evidence stage: completed the self-rating but declined evidence review, refused the selected dataset, and produced partial evidence. These are the outcomes most likely to correlate with low maturity.

Publish bounds as well as point estimates. A sensitivity analysis that assigns level 0 to every eligible company that declined the evidence stage gives a floor; the observed distribution among completers gives a ceiling. If weights are used to adjust for size band or sector, publish the weighting variables, the source of population totals and both weighted and unweighted results.

Pre-registration checklist before the first invitation goes out

Register the protocol in a public, time-stamped place before the first invitation and keep a log of every later change with its date and reason.

At a glance

Pre-registration checklist before the first invitation goes out

Checklist · 13 items

  1. Research questions, ranked, with the conclusions the design cannot support.
  2. Population definition, eligibility screen and the date eligibility is measured.
  3. Sampling frame, its known gaps, strata and target counts per stratum.
  4. Within-company dataset definition, the random selection method and the rule for refusals.
  5. Instrument version: domains, levels, rubric with examples and the applicability rule.
  6. Self-rating questions and the order in which self-rating and evidence review happen.
  7. Reviewer training, double-scoring share and the disagreement rule.
  8. Disposition codes, the response rate formula and the method for estimating eligibility among unknown cases.
  9. Primary and secondary metrics, interval method, weighting plan and sensitivity analyses.
  10. Minimum cell size for publication and the rule for suppressing small strata.
  11. Data handling: where evidence is viewed, what is retained, and when it is destroyed.
  12. Fieldwork dates, reminder schedule, incentive policy and funding sources.
  13. Planned publication date and a commitment to publish whatever the results show.

0 of 13 ticked. Tick each item as you go.

Confidentiality, reviewer conduct and what gets published

Reviewers need to see contracts, policies and samples of records, so the study needs confidentiality terms that let companies show evidence without handing it over. A practical approach is review on the company's own systems or in a shared session, with the reviewer recording only the level and a short evidence note.

Publish aggregates only, with no company or dataset named and small strata suppressed below the pre-registered minimum. Keep the reviewer notes separate from contact details. Make clear to participants that a level score is a research measurement of documentation, not a certification, audit opinion or legal view.

Regulation gives the work context without changing the design. The EU AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used for training, following a template from the AI Office. A survey of internal evidence measures something narrower and more granular than that public summary, and the write-up should say so rather than suggest compliance findings.

Using the instrument on your own records before a licensing review

A company can run the instrument on its own datasets before any external study, as a self-check of what it could show a buyer. Pick a dataset at random rather than the best one, score the eight domains honestly, and treat any domain stuck at level 1 as the first piece of work.

For a business that later licenses records directly to SourceX, the rights, personal data and lineage domains above line up with the rights review and data preparation SourceX carries out before contracting. Data moves only after the business has approved scope and terms and signed, and the opening step is a brief company assessment that does not ask for any dataset.

Frequently asked questions

What is a dataset provenance maturity survey?

It is a study that measures how well organizations can document where their datasets came from, under what terms and how they were processed. The evidence-based version scores documents a reviewer actually inspects for one sampled dataset per company, rather than relying on self-ratings.

Does this page report survey results?

No. It publishes the protocol only. No survey has been run under this design, and any result would need the sampling, response rates and uncertainty described here before it could be reported.

Why sample one dataset per company at random?

Because companies asked to choose will show their best-documented dataset, which overstates maturity. Random selection from a list the company provides, with a logged seed and no substitutions, gives an estimate that reflects typical datasets.

Can a provenance score show that data may be licensed?

No. The score shows whether documents exist and match the records. Whether a particular use or license is lawful depends on contracts, notices and the applicable law, which counsel assesses on the specific facts.

What population would a defensible study cover?

A defined set of companies, by size, country and data role, drawn from a named sampling frame with known gaps. Results apply only to that population, and the write-up should say what the frame misses.

How should nonresponse be handled?

Record a disposition for every invited company, compute response rates with a published formula, and report bounds that assume low maturity for eligible companies that declined the evidence stage. Companies with weak records are the likeliest to decline.

Sources

  • The W3C PROV-DM defines provenance as information about the entities, activities and people involved in producing data, used to assess its quality, reliability or trustworthiness. Source
  • The Data and Trust Alliance describes its Data Provenance Standards as cross-industry metadata standards for the origin of datasets used in data and AI applications. Source
  • NIST AI 600-1 suggests documenting the origin of training data, including copyright, licensing and privacy status, and keeping audit trails of sources and transformations. Source
  • The EU AI Act requires general-purpose AI model providers to publish a sufficiently detailed summary of training content using an AI Office template. Source
  • The Data Provenance Initiative audited 44 data collections spanning more than 1,800 fine-tuning datasets, documenting sources, licenses and creators. Source
  • Datasheets for Datasets proposes that each dataset be documented with its motivation, composition and collection process. Source
  • AAPOR's Standard Definitions classify survey cases into four disposition groups, define a response rate as completes over eligible units, and warn against choosing eligibility estimates to inflate response rates. Source
  • AAPOR published Standard Definitions for Establishment Surveys in December 2025. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies