AI data market
How to run a dataset provenance maturity survey: the study protocol
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A study can assess whether companies can prove where their training data came from by asking each sampled company to show documents for one randomly chosen dataset, then scoring verified evidence for origin, terms, rights, processing, personal data and delivery. Report levels with intervals, response rates and the self-rating gap. This page is the protocol only; it reports no results.
Provenance trail
Dataset provenance maturity
Key takeaways
- Measure evidence a reviewer inspects, not self-rated maturity.
- Sample one dataset per company at random, with no substitutions, so participants cannot show only their best-documented data.
- Score eight domains on four levels and let the weakest core domain set the overall level.
- Record a disposition for every invited company and publish bounds that assume low maturity for those who decline.
- Pre-register questions, frame, instrument, metrics and the commitment to publish; no results are reported here.
How could a study assess whether companies can prove where their training data came from?
Ask each company to show documents for one randomly chosen dataset, not to rate itself. A reviewer then scores whether evidence exists, and holds up on a sample of records, for origin, collection terms, rights, transformations, personal data handling and delivery. The study reports the share of datasets at each level with intervals, response rates and the gap between self-rating and verified evidence.
This page is a research protocol, not a report of findings. No survey has been run under it, and no results, percentages or rankings appear here. What follows is the design: the question, population, sampling, instrument, variables, comparison, metrics, uncertainty, limits and a pre-registration checklist that a research team, trade body or journalist could adopt and publish before collecting any data.
The reason to insist on evidence is simple. Self-assessed maturity surveys measure confidence, and confidence about data origins is easy to state and hard to check. A design that inspects the documents behind one dataset per company measures what a buyer, auditor or regulator would actually be shown.
What the survey counts as provenance, and what it leaves to lawyers
The survey uses the W3C definition: provenance is information about the entities, activities and people involved in producing a piece of data, which can be used to judge its quality, reliability or trustworthiness. That frames the measurement as a question about records that exist, not about whether a use is lawful.
Four things are kept apart in the instrument. Access is whether someone can technically export the data. Privacy permission is whether personal information in it may be used for a given purpose. Ownership is who holds title to the records or the intellectual property in them. Contractual licensing rights are what the company may grant to others. De-identifying a dataset does not by itself create a right to license it, so the instrument scores rights evidence separately from privacy evidence.
Several published frameworks name what provenance evidence should cover, and the instrument borrows their categories rather than inventing new ones. The Data and Trust Alliance describes its Data Provenance Standards as cross-industry metadata standards for the origin of datasets used in traditional data and AI applications. NIST's Generative AI Profile (NIST AI 600-1) suggests actions such as documenting the origin of training data, including its copyright, licensing and privacy status, and keeping audit trails of sources and transformations. The Datasheets for Datasets proposal by Gebru and colleagues sets out questions about a dataset's motivation, composition and collection process.
Prior audits show the method is feasible on public data. The Data Provenance Initiative documented sources, licenses and creators across 44 data collections spanning more than 1,800 fine-tuning datasets. That audit covered publicly released datasets; this protocol targets something different, the internal records companies hold about data they own or license.
Research questions and the conclusions this design cannot support
State the questions before recruiting anyone, and state the conclusions the design will not support with the same prominence. Both belong in the pre-registration.
At a glance
Research questions and the conclusions this design cannot support
- Primary question: among sampled datasets held by companies in the defined population, what share has verified evidence at each maturity level in every core domain?
- Secondary question: how far do companies' self-ratings differ from the verified level for the same dataset and domain?
- Secondary question: which domains most often set the overall level, meaning they are the weakest link?
- Exploratory only: do levels differ by company size band, sector or data role (supplier of records versus developer training models)?
- Not supported: claims about companies outside the sampling frame, about all datasets a company holds, or about causes of maturity differences.
- Not supported: any judgment that a dataset may lawfully be used or licensed. The reviewer scores the existence and consistency of documents; counsel decides legal sufficiency on specific facts.
Who is in the population, and how one dataset per company is drawn
Define the population as companies, not datasets, then draw one dataset per company at random from a list the company provides. This prevents the most common bias in provenance studies, where each participant shows its best-documented dataset.
A workable definition is companies above a stated employee threshold, in named countries, that either hold business records they have licensed or are considering licensing for AI, or train or fine-tune models on data they did not create. Write the exact eligibility screen into the protocol, including how employee counts are measured and on what date.
No complete list of such companies exists, so the sampling frame will be imperfect. Options include business registers filtered by size and sector, membership lists of industry associations, and public lists of AI developers. Record which frame is used, what it is known to miss, and stratify by size band and sector so small strata are not lost. Recruitment dates, invitation wording, the number of reminders and any non-cash incentive go in the protocol before the first contact.
Within each participating company, ask for a list of datasets that meet a stated definition (for example, a dataset prepared for or used in model training in a stated period), number them, and select one with a random number generator whose seed is logged. If the company refuses the selected dataset, record a refusal at that stage instead of accepting a substitute.
Size the sample for precision, not for a hoped-for result. For a proportion, the half-width of a 95% interval is at most about 1.96 times the square root of 0.25 divided by n. As planning arithmetic only, n of 400 completed cases gives a worst-case half-width near 4.9 percentage points; strata will be wider. Adjust for expected eligibility and response before setting invitation counts.
The evidence-based provenance maturity instrument
The instrument scores eight domains at four levels. Level 0 means no evidence; level 1 means the company asserts the fact but produces nothing; level 2 means a document exists and is produced; level 3 means the document is produced and checks out against a sample of the dataset's records. The table defines levels 1 to 3 and the evidence the reviewer inspects.
| Domain | Level 1: asserted | Level 2: documented | Level 3: verified on a record sample | Evidence the reviewer inspects |
|---|---|---|---|---|
| Origin systems and dates | Names the source systems | System list with export dates and record counts | Sampled records trace to a named system and date range | Export logs, system inventory, record IDs |
| Collection terms and notices | States users were informed | Notice or terms versions with effective dates | Each sampled record falls under a produced version | Policy versions, collection timestamps |
| Rights chain | Claims ownership or license | Contracts, licenses or assignments produced | Sampled records fall within the produced grants | Agreements, IP assignments, scope clauses |
| Third-party content | States none is included | Screening method written down | Sample screened with results logged | Screening log, exclusion list |
| Personal data handling | States data is de-identified | Method and responsible party documented | Sample checked against the stated method | Method note, test log, exceptions |
| Transformations and lineage | Describes processing loosely | Step list with versions and dates | Sample reproduced from source through the steps | Pipeline config, version tags, diffs |
| Version and delivery | Says a copy was sent | Manifest with file list and counts | Hashes or counts match the delivered copy | Manifests, checksums, delivery receipts |
| Dataset documentation | Informal description | Datasheet or equivalent completed | Datasheet statements match the sample | Datasheet, data dictionary |
Scoring rules: why the weakest domain sets the level
The overall level for a dataset is the lowest level reached in any of the six core domains, not the average. A dataset with verified origin and lineage but no rights evidence cannot be licensed with confidence, and an average would hide that.
Core domains are origin, collection terms, rights chain, personal data handling, transformations and version and delivery. Third-party content and documentation are reported separately because they do not apply equally to every dataset type; write the applicability rule into the protocol.
Two reviewers score a pre-set share of cases independently, chosen at random, and the study reports agreement by domain with a chance-corrected statistic. Disagreements go to a third reviewer, and the rule for that is fixed in advance. Reviewers score from a written rubric with examples of each level and see the company's self-rating only after scoring.
The self-rating is collected first, in the same eight domains on the same four-level scale, so the comparison between claimed and verified levels uses identical categories. Report the gap as the distribution of differences, not just a mean.
| Metric | Definition | Unit of analysis | Uncertainty reported |
|---|---|---|---|
| Verified level distribution | Share of datasets at overall levels 0 to 3 | Dataset | 95% interval per level, weighted and unweighted |
| Weakest-link domain | Domain that sets the overall level | Dataset | Counts with intervals; ties reported |
| Self-rating gap | Self-rated level minus verified level, per domain | Dataset and domain | Distribution and interval for the median |
| Reviewer agreement | Chance-corrected agreement on double-scored cases | Domain | Point estimate with interval |
| Completion by stage | Share completing screen, self-rating and evidence review | Company | Counts at each stage |
Response rates, uncertainty and the companies that never answer
Report every invited company in a disposition, and calculate response rates from those dispositions, because the companies least able to show evidence are the most likely to decline. The American Association for Public Opinion Research (AAPOR) Standard Definitions sort cases into completed interviews, eligible nonrespondents, cases of unknown eligibility and ineligible cases, and define a response rate as completes divided by eligible units.
AAPOR also warns that the estimate of how many unknown-eligibility cases are actually eligible must rest on the best available evidence and must not be chosen to raise the response rate. In December 2025 it published Standard Definitions for Establishment Surveys, which is the closer fit for a survey of companies, and the protocol should name which edition it follows.
Add a second, study-specific disposition layer for the evidence stage: completed the self-rating but declined evidence review, refused the selected dataset, and produced partial evidence. These are the outcomes most likely to correlate with low maturity.
Publish bounds as well as point estimates. A sensitivity analysis that assigns level 0 to every eligible company that declined the evidence stage gives a floor; the observed distribution among completers gives a ceiling. If weights are used to adjust for size band or sector, publish the weighting variables, the source of population totals and both weighted and unweighted results.
Pre-registration checklist before the first invitation goes out
Register the protocol in a public, time-stamped place before the first invitation and keep a log of every later change with its date and reason.
At a glance
Pre-registration checklist before the first invitation goes out
Checklist · 13 items
- Research questions, ranked, with the conclusions the design cannot support.
- Population definition, eligibility screen and the date eligibility is measured.
- Sampling frame, its known gaps, strata and target counts per stratum.
- Within-company dataset definition, the random selection method and the rule for refusals.
- Instrument version: domains, levels, rubric with examples and the applicability rule.
- Self-rating questions and the order in which self-rating and evidence review happen.
- Reviewer training, double-scoring share and the disagreement rule.
- Disposition codes, the response rate formula and the method for estimating eligibility among unknown cases.
- Primary and secondary metrics, interval method, weighting plan and sensitivity analyses.
- Minimum cell size for publication and the rule for suppressing small strata.
- Data handling: where evidence is viewed, what is retained, and when it is destroyed.
- Fieldwork dates, reminder schedule, incentive policy and funding sources.
- Planned publication date and a commitment to publish whatever the results show.
0 of 13 ticked. Tick each item as you go.
Confidentiality, reviewer conduct and what gets published
Reviewers need to see contracts, policies and samples of records, so the study needs confidentiality terms that let companies show evidence without handing it over. A practical approach is review on the company's own systems or in a shared session, with the reviewer recording only the level and a short evidence note.
Publish aggregates only, with no company or dataset named and small strata suppressed below the pre-registered minimum. Keep the reviewer notes separate from contact details. Make clear to participants that a level score is a research measurement of documentation, not a certification, audit opinion or legal view.
Regulation gives the work context without changing the design. The EU AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used for training, following a template from the AI Office. A survey of internal evidence measures something narrower and more granular than that public summary, and the write-up should say so rather than suggest compliance findings.
Using the instrument on your own records before a licensing review
A company can run the instrument on its own datasets before any external study, as a self-check of what it could show a buyer. Pick a dataset at random rather than the best one, score the eight domains honestly, and treat any domain stuck at level 1 as the first piece of work.
For a business that later licenses records directly to SourceX, the rights, personal data and lineage domains above line up with the rights review and data preparation SourceX carries out before contracting. Data moves only after the business has approved scope and terms and signed, and the opening step is a brief company assessment that does not ask for any dataset.
Frequently asked questions
What is a dataset provenance maturity survey?
It is a study that measures how well organizations can document where their datasets came from, under what terms and how they were processed. The evidence-based version scores documents a reviewer actually inspects for one sampled dataset per company, rather than relying on self-ratings.
Does this page report survey results?
No. It publishes the protocol only. No survey has been run under this design, and any result would need the sampling, response rates and uncertainty described here before it could be reported.
Why sample one dataset per company at random?
Because companies asked to choose will show their best-documented dataset, which overstates maturity. Random selection from a list the company provides, with a logged seed and no substitutions, gives an estimate that reflects typical datasets.
Can a provenance score show that data may be licensed?
No. The score shows whether documents exist and match the records. Whether a particular use or license is lawful depends on contracts, notices and the applicable law, which counsel assesses on the specific facts.
What population would a defensible study cover?
A defined set of companies, by size, country and data role, drawn from a named sampling frame with known gaps. Results apply only to that population, and the write-up should say what the frame misses.
How should nonresponse be handled?
Record a disposition for every invited company, compute response rates with a published formula, and report bounds that assume low maturity for eligible companies that declined the evidence stage. Companies with weak records are the likeliest to decline.
Sources
- The W3C PROV-DM defines provenance as information about the entities, activities and people involved in producing data, used to assess its quality, reliability or trustworthiness. Source
- The Data and Trust Alliance describes its Data Provenance Standards as cross-industry metadata standards for the origin of datasets used in data and AI applications. Source
- NIST AI 600-1 suggests documenting the origin of training data, including copyright, licensing and privacy status, and keeping audit trails of sources and transformations. Source
- The EU AI Act requires general-purpose AI model providers to publish a sufficiently detailed summary of training content using an AI Office template. Source
- The Data Provenance Initiative audited 44 data collections spanning more than 1,800 fine-tuning datasets, documenting sources, licenses and creators. Source
- Datasheets for Datasets proposes that each dataset be documented with its motivation, composition and collection process. Source
- AAPOR's Standard Definitions classify survey cases into four disposition groups, define a response rate as completes over eligible units, and warn against choosing eligibility estimates to inflate response rates. Source
- AAPOR published Standard Definitions for Establishment Surveys in December 2025. Source
See if your company qualifies
A short company assessment. No data uploads are needed.