AI data market
Dataset provenance utility evaluation: a protocol for two outcomes
By SourceX Editorial · Updated
10 sections
Short answer
To test whether better provenance speeds buyer review or improves model utility, run two preregistered experiments that share one fixed provenance package. Reviewers assess the same dossiers with and without it, measured as time to a correct decision. Models are trained on data selected with and without provenance at equal size and budget, scored on an untouched holdout. The page sets out that method and reports no findings.
Evaluation split
Dataset provenance utility
Key takeaways
- Treat the question as two experiments, because provenance affects review and model utility through different mechanisms.
- Define better provenance as fixed, versioned packages built from published vocabularies, plus a placebo matched in length.
- Measure review as time to a correct decision on dossiers seeded with known issues, not raw speed.
- Test model utility by comparing provenance-informed selection with blind selection and a shuffled-label control at equal size and budget.
- Preregister one primary metric per arm, the sample size and the interval method; this page reports no results.
How do you test whether better provenance speeds review or improves a model?
Run two linked, preregistered experiments that share one provenance treatment. The procurement arm gives reviewers the same dataset dossiers with and without a defined provenance package and measures time to a correct decision. The task arm trains or tunes the same model on data selected with and without that provenance, under equal budgets, and scores both on an untouched holdout and a business task.
This page publishes the method only. It reports no results, because none have been collected under this design, and it should not be read as evidence that provenance does or does not help. What it gives you is a design two teams could run independently and compare.
The two outcomes need separate arms because they work through different mechanisms. Provenance can shorten review because reviewers stop chasing answers the package already holds. It can change model utility only indirectly, through which records are kept, removed, deduplicated or weighted. Mixing the two into one score hides which mechanism, if either, did the work.
What exactly is the treatment when you say better provenance?
Better provenance has to be a fixed, versioned package with named fields, not a general impression that one dossier is more thorough. If the treatment is vague, any difference you measure could come from formatting, length or the care someone took assembling one dossier.
Build the package levels from published vocabularies so another team can reproduce them. The Croissant 1.1 metadata specification recommends the W3C PROV Ontology for provenance and names three core properties: wasDerivedFrom for an entity derived from another, wasGeneratedBy for the activity that produced it, and wasAttributedTo for the responsible agent. Its Responsible AI extension adds data life cycle details such as sources, creation dates, versions and processing.
Datasheets for Datasets supplies the narrative layer (motivation, composition, collection process and recommended uses), and the Data & Trust Alliance publishes cross-industry provenance metadata standards built from industry use cases. Pick one level per arm and freeze it before recruitment.
| Level | What the dossier contains | Typical source of the fields | What it lets a reviewer or pipeline do |
|---|---|---|---|
| P0 baseline | File manifest, row counts, schema and a one-paragraph description | Whatever the supplier sends by default | Open the data and guess at its origin |
| P1 dataset level | P0 plus generating activity, source systems, collection dates, responsible party and license or agreement reference | PROV-O properties as used in Croissant, plus a datasheet | Answer most origin questions without emailing the supplier |
| P2 record level | P1 plus a source identifier, derivation link and permitted-use flag on every record | Record-level fields exported from the source system | Filter, deduplicate or stratify individual records by origin |
| Placebo | P0 padded to the same length as P1 with neutral, accurate but non-provenance text | Editorial filler checked by the study lead | Tests whether length and polish alone change behavior |
Outcome one: does a richer provenance package change buyer review?
Measure review as time to a correct decision, not time to any decision. A reviewer who approves a dataset quickly while missing a planted license conflict has not been helped; the package has made the review faster and worse.
The population is people who actually review data acquisitions: legal or privacy counsel, data leads and procurement staff. The unit is one reviewer reviewing one dossier. Use a crossover design: each reviewer sees several dossiers, some in the baseline condition and some in the treatment condition, with dossier, condition and order randomized and counterbalanced so no dossier is always seen first or always with provenance.
Seed every dossier with known issues before the study, such as a source whose terms bar commercial use, a derived copy of another source, or a missing collection date. The Data Provenance Initiative's audit of more than 1,800 text datasets found license omissions above 70% and error rates above 50% on popular hosting sites, which is why the seeded issues should include wrong labels, not only missing ones.
Record the clock from dossier open to a submitted decision, the decision itself, which seeded issues were found, every clarification question the reviewer would have sent, and a confidence rating. Cap review time and treat unfinished reviews as censored observations rather than dropping them.
Outcome two: can provenance change model utility at all?
Provenance changes a model only by changing the training data, so the task arm tests provenance-informed selection against selection without it. Hold the candidate pool, the final record count, the model, the training recipe and the compute budget fixed, and vary only what the selection step is allowed to see.
Run at least three conditions. Condition A selects records by the usual quality filters with provenance hidden. Condition B uses the same filters plus provenance: it removes records with unknown or barred origin, collapses derived copies of one source, and stratifies by source system. Condition C repeats B with the provenance fields shuffled across records, which keeps the selection machinery and the data size but breaks the link between label and record.
Give each condition the same hyperparameter search budget. Dodge and colleagues showed that model comparisons can reverse when one side gets more search trials or training time, so report expected performance against budget rather than one best run.
Score every condition on a holdout built and frozen before any selection runs, and on a business outcome task drawn from the buyer's real use, such as routing a work order or extracting fields from an invoice. Build the holdout from sources that never enter the candidate pool, then check that no derived copy slipped in; provenance itself is the best tool for that check, so run it with P2 fields in every condition.
What does the full two-arm design commit to in advance?
This table is the asset: the full design, with what each arm commits to and what must be written into the registration before any data is seen. Copy it into the registration form and fill the last column with your own values.
At a glance
What does the full two-arm design commit to in advance
| Design element | Procurement arm | Task arm | Write into the registration |
|---|---|---|---|
| Research question | Does provenance level P1 or P2 shorten time to a correct acquisition decision compared with P0? | Does provenance-informed selection change holdout and business-task scores compared with selection without provenance? | The exact question, the direction you expect and the smallest effect you would act on |
| Population and unit | Working data reviewers; one reviewer by one dossier | One candidate data pool; one training run per condition and seed | Inclusion rules for reviewers and for records in the pool |
| Sampling | Recruit across legal, data and procurement roles; stratify assignment by role | Fixed pool; resample the selected set several times per condition | Recruitment channel, strata and resampling scheme |
| Treatment and control | P0, P1 or P2 dossier, plus the placebo dossier | Conditions A, B and C at equal final record count | The frozen dossier versions and selection code commit |
| Randomization | Dossier, condition and order randomized and counterbalanced | Seeds and resamples drawn from a recorded generator | Who generates the assignment and when it is unsealed |
| Primary metric | Time to a correct decision, with censoring at the time cap | Holdout score on the pre-declared task metric | One primary metric per arm, nothing else |
| Secondary metrics | Seeded issues found, clarification questions, confidence | Business-task score, regressions on other tasks, variance across seeds | The full list, labeled secondary |
| Uncertainty | Intervals that resample reviewers and dossiers | Intervals across seeds and data resamples | The interval method and the level reported |
| Stopping rule | Fixed number of reviewers set from a pilot | Fixed number of seeds and resamples set from a pilot | Sample size, how it was chosen, no peeking rules |
| Exclusions | Reviewer with a technical failure or prior exposure to a dossier | Run with an infrastructure fault, logged before scores are read | Exclusion criteria fixed in advance |
Which alternative explanations could fake an improvement?
Most apparent wins in a study like this come from something other than provenance, so each one needs a design control written in before the study starts. The table lists the likely ones and the follow-up check that rules each out.
| Alternative explanation | How it shows up | Design control |
|---|---|---|
| Learning across dossiers | Reviewers get faster on later dossiers regardless of condition | Counterbalance order and include order as a model term |
| Length or polish, not provenance | Any longer, tidier dossier is reviewed with more confidence | The placebo dossier matched in length and layout |
| Reviewer expertise imbalance | Senior reviewers cluster in one condition | Within-reviewer crossover and stratified assignment by role |
| Different data size | Provenance filtering leaves fewer or more records | Equal final record count across conditions A, B and C |
| Unequal tuning effort | The provenance condition got extra search trials | One search budget for every condition, reported as a curve |
| Selection machinery, not labels | Any extra filter step helps by chance | Condition C with shuffled provenance fields |
| Holdout leakage | Derived copies of holdout items sit in the training pool | Holdout built from excluded sources and checked with record-level derivation links |
| Seed and sampling noise | One lucky run carries the difference | Several seeds and resamples per condition, all runs reported |
| Flexible analysis | The metric that moved is reported, the others are dropped | One preregistered primary metric per arm |
What sample, baseline and uncertainty make a finding defensible?
A finding is defensible when the comparison is against a real baseline, the sample was fixed before the study, and every estimate carries an interval built from the sources of variance that actually exist. A single number from one run, or an average over hand-picked reviewers, is not.
Set the sample from a pilot. Run a small version of each arm, estimate the spread of the primary metric, and choose the number of reviewers, seeds and resamples needed to detect the smallest effect you said you would act on. Write that reasoning into the registration so a reader can check it.
Treat model scores as random quantities. Bouthillier and colleagues found that data sampling, parameter initialization and hyperparameter choices each change benchmark results noticeably, and recommend randomizing as many of those sources as is feasible. In the task arm, resample the selected data as well as the seed, because a provenance filter can look good on one draw of records and neutral on another.
For the procurement arm, compute intervals that resample both reviewers and dossiers, since a result that holds only for one dossier is a property of that dossier. Report censored reviews with a time-to-event method rather than averaging only the reviews that finished. Report all conditions, including the placebo, even if they look uninteresting.
What goes in the registration before the first dossier opens?
File the registration before anyone opens a dossier or runs a selection. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of the study plan to a public repository before data collection or analysis, and asks that exploratory analyses be clearly separated from confirmatory ones in the report. A private, dated document held by an independent person works for internal studies, provided it cannot be edited unnoticed.
- The research question for each arm, the expected direction and the smallest effect that would change a decision.
- The provenance levels, the placebo dossier and the selection code, each frozen by version or commit.
- Reviewer inclusion rules, recruitment strata and the randomization procedure, including who holds the assignment.
- The seeded issues in each dossier and the answer key, sealed until scoring.
- The candidate pool, the holdout sources, the business task and the leakage check run before training.
- One primary metric per arm, the secondary metrics, and the time cap and censoring rule.
- The sample size and the pilot that set it, plus exclusion criteria.
- The interval method and the comparisons you will report, including the placebo and condition C.
- A statement that results will be published whatever their direction, with any deviations listed and explained.
What can this protocol not answer, even when it runs cleanly?
Even a clean result answers a narrow question: whether this provenance package, for these dossiers, these reviewers and this model, changed the measured outcomes. It does not show that provenance helps on a different data type, and it does not put a value on provenance outside the study.
Provenance is also evidence, not permission. A package can show where every record came from and still show that a source cannot be licensed for training; de-identification of those records would not change that. Rights questions go to counsel with the actual agreements, whatever the experiment finds.
Finally, the procurement arm measures the review step only. Time saved there can be offset by time spent assembling record-level provenance upstream, so if that cost matters to your decision, log supplier preparation hours as a separate outcome rather than inferring it from review speed.
Can the protocol run on licensed business records?
Both arms need a dataset whose provenance is known down to the record, which is easier when the data comes straight from a company's own operating systems with a signed license. SourceX can supply data for either arm only by request: a study team lists its scope, volume, language, format, timing and licensing needs on the request-data form, and SourceX then sources the records from U.S. businesses instead of drawing on a held inventory.
Each supplying company signs its agreement, after approving the scope and terms, before a single record reaches SourceX, which gives the study's provenance package a dated starting document instead of a history pieced together later.
Frequently asked questions
What is dataset provenance utility evaluation?
It is a controlled test of whether provenance information about a dataset changes something you care about, such as how quickly buyers reach a correct decision or how well a model trained on the data performs. The protocol here compares fixed provenance packages against a baseline and a placebo. It reports no results.
Can better provenance make a model more accurate by itself?
No. A model never sees the provenance package, so any effect has to come through the data: records removed for unknown origin, derived copies collapsed, or sources rebalanced. That is why the task arm compares selection with and without provenance at equal data size and budget.
How do you measure buyer review speed fairly?
Measure time to a correct decision, not time to any decision. Seed dossiers with known issues, randomize and counterbalance which dossiers each reviewer sees with and without provenance, cap review time and treat unfinished reviews as censored.
Why include a placebo dossier?
A longer, tidier dossier may feel more trustworthy whatever it contains. A placebo matched in length and layout but without provenance fields separates the effect of provenance from the effect of presentation.
Are there published results showing provenance speeds data review?
This page does not report any, and its sources do not include a controlled study of review speed. The Data Provenance Initiative documented widespread license omissions and errors on dataset hosting sites, which motivates the question but does not answer it.
Sources
- The Croissant 1.1 specification recommends the W3C PROV Ontology for dataset provenance and lists wasDerivedFrom, wasGeneratedBy and wasAttributedTo as its core properties. Source
- The Croissant Responsible AI specification describes data life cycle information such as creators, creation dates, sources, versions and processing. Source
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- The Data & Trust Alliance publishes cross-industry metadata standards for the origin of datasets, developed from use cases across industries. Source
- The Data Provenance Initiative audited more than 1,800 text datasets and found license omission above 70% and error rates above 50% on popular dataset hosting sites. Source
- Reporting expected validation performance as a function of computation budget shows comparisons whose conclusion changes with more or less computation. Source
- Variance from data sampling, parameter initialization and hyperparameter choice noticeably affects benchmark results, and the authors recommend randomizing as many sources of variation as possible. Source
- Preregistration is posting a time-stamped, read-only version of a study plan to a public repository before beginning data collection or analysis. Source
- Exploratory analyses should be clearly differentiated from confirmatory analyses in the report. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.