Skip to content

Privacy and preparation

Dataset provenance evidence hierarchy: how strong is each kind of proof?

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Rank provenance evidence by how it came to exist, not how official it looks. System records created at the time of the event and bound to the delivered files by checksums are strongest; executed agreements and versioned notices come next for authority and permission; pipeline run records prove transformations; reconstructions, declarations and recollection are weakest. Grade each claim by its weakest essential link and log anything below the bar in an unresolved gaps register.

Provenance trail

Dataset provenance evidence

This page covers dataset provenance evidence hierarchy: how strong is each kind of proof.Illustrative

Key takeaways

  • Grade claims, not documents: source, transformation, authority, permission and release each need their own evidence.
  • Four tests rank any item: timing, whether a system or a person produced it, binding to exact records, and independence.
  • A matching checksum proves a file is unchanged; it does not prove the content is accurate or licensable.
  • A claim is only as strong as its weakest essential link, and grades should never be averaged across a dataset.
  • Weak claims go on an unresolved gaps register with scope, owner, disposition and the evidence that would close them.

How do you assess the strength of different forms of provenance evidence?

Judge each piece of provenance evidence by four tests: when it was made relative to the event it describes, whether a system or a person produced it, whether it is bound to the exact records and release in question, and whether it is independent of the party making the claim. Evidence that passes all four sits at the top of the hierarchy; evidence that passes none is a lead to investigate, not proof.

These tests are borrowed from two fields that have ranked evidence for decades. The U.S. Federal Rules of Evidence give special weight to a record made at or near the time of an event, by someone with knowledge, as a regular practice of an ongoing activity, and still let a court exclude it when the source or circumstances suggest it is untrustworthy. The PCAOB audit evidence standard, AS 1105, says reliability depends on the nature and source of evidence and the circumstances in which it was obtained, and ranks original documents above copies or digitized versions, whose reliability depends on controls over the conversion.

Neither rule governs AI datasets. They are useful because they separate how convincing a document looks from how it came to exist, which is exactly the distinction a dataset reviewer needs when a polished summary sheet and a raw event log say different things.

Which dataset claims need their own grade?

Grade claims, not documents. One export manifest might strongly support a claim about what was delivered and say nothing about whether the company could license it. Split every provenance statement in a datasheet, attestation or data room index into five claim types and grade each separately.

At a glance

Which dataset claims need their own grade

The W3C PROV-O ontology gives a vocabulary for the first three: entities (records and files), activities (extraction, redaction, labeling) and agents (the people, systems and companies responsible), linked by relations such as wasGeneratedBy, wasDerivedFrom and wasAttributedTo. Authority and permission sit outside that model because they are legal conclusions drawn from documents, and they need counsel as well as evidence. The NIST AI RMF Playbook asks a similar question of AI developers: how the data's provenance is documented, including sources, origins, transformations, augmentations, labels, dependencies, constraints and metadata.

  • Source: these records came from this system, for this period, from this business unit.
  • Transformation: these steps, at these versions, turned source records into the delivered rows.
  • Authority: this entity held the records and could decide to license them.
  • Permission: the notices, customer terms and agreements in force when each record was created allow this use; access to a system, privacy permission, ownership and a contractual right to license are four different questions.
  • Release: the files delivered are the files described, unchanged since they were checked.

The dataset provenance evidence hierarchy

The table ranks the evidence forms a data owner usually holds, from strongest to weakest, with what each can and cannot carry. Tier numbers are an editorial method for ranking, not an industry standard, and a reviewer can move an item between tiers using the factors in the next section.

Read the last two columns together. Most disputes about provenance come from evidence being stretched past what it can support: a clean export manifest offered as proof of permission, or a signed customer contract offered as proof that a particular row is covered. Before citing any item, name the claim type it supports, check that the claim falls inside the can-support column, and look for a second item to cover whatever sits in the cannot-support column. If no second item exists, the claim belongs on the gaps register rather than in the datasheet.

The dataset provenance evidence hierarchy
TierEvidence formExamples from business systemsWhat it can supportWhat it cannot support
1Contemporaneous system records bound to the releaseTicket history or field change logs from the source application, exported with a manifest listing each file and its SHA-256 checksumSource and release claims for the exact records listedWhether anyone had the right to license those records
2Executed or versioned authority documentsSigned customer agreements, a board or officer approval, a privacy notice archived with its effective dateAuthority and permission claims, for the period and parties the document coversThat a given record falls inside that period or party list, until linked by date or account ID
3Pipeline run recordsTransformation code at a tagged commit, run logs with input and output row counts, redaction rule versionsTransformation claims for the run that produced a stated releaseBehavior of earlier or later runs, or of manual edits outside the pipeline
4Reconstructed recordsA re-export made months later, a mapping table rebuilt from memory of old fields, screenshots of settingsLeads and partial corroboration when tier 1 to 3 records are lostWhat the system contained or how it was configured on the original date
5DeclarationsAttestations, vendor questionnaire answers, a signed statement from a department headWho is accountable for a claim and on what dateThe underlying facts; a declaration is only as good as the records behind it
6Recollection and inferenceInterviews with former administrators, statements such as we never stored card numbers in notesWhere to look for better evidenceAny claim on its own

What moves a piece of evidence up or down a tier?

Binding to the exact bytes moves evidence up. The committee notes to Federal Rule of Evidence 902(14) say copied electronic data is ordinarily authenticated by hash value, because matching hashes show an original and a copy are identical. The same notes are clear that this proves authenticity only. A matching checksum shows the delivered file is the file that was checked; it says nothing about whether the content is accurate or licensable.

Who generated the record matters as much as when. The SLSA supply chain framework draws this line for software: at its first build level, provenance exists but is trivial to bypass or forge; at the second, a hosted build platform generates and signs it, so forging it takes a deliberate attack. A dataset run log written by a scheduled pipeline service under its own credentials deserves more weight than a spreadsheet a team member filled in after the fact, even when the two say the same thing.

Several conditions move evidence down: gaps in sequence numbers or timestamps, edits made after the event, records produced only by the party that benefits from the claim, and copies whose conversion was not controlled. AS 1105 treats evidence from knowledgeable sources outside the company as generally more reliable than evidence from inside it, so a payment processor's statement or a customer's countersigned order form can corroborate an internal log in a way a second internal report cannot.

How should conflicting or partial evidence be graded?

A claim takes the grade of its weakest essential link. If the source is proven at tier 1 but the permission rests on a tier 5 declaration, the claim the buyer cares about, that these records may be licensed for training, is a tier 5 claim until the permission evidence improves.

When two items conflict, the higher tier wins only if both speak about the same records and the same release. A tier 2 notice that took effect in March does not override a tier 1 log showing records created in January; it simply does not cover them. Never average grades across a dataset. Split it instead: records whose essential links are strong form one slice, and the rest go on the gaps register with a clear statement of what cannot yet be said about them.

Live and append-only datasets need one more rule. The Croissant metadata format, maintained by MLCommons, recommends recording a checksum for every file in a versioned dataset but not for files still being updated, and it treats changes to past records as a version change. Grade a growing dataset per closed period, and treat any later edit to a closed period as a new release that needs fresh tier 1 evidence.

Can a reviewer trace a graded claim to the exact record and release?

Only if every delivered row carries a small set of pointers back to its evidence. Without them, a strong manifest and a strong contract sit side by side without proving anything about a particular row, and the reviewer is back to trusting a summary.

Keep the pointers short and store the evidence once. A row does not need a copy of the customer agreement; it needs the identifier of the agreement version that applied on the date the record was created. The same idea applies to transformations: one run record describes thousands of rows, so each row carries the run ID rather than a description of what the run did.

Test the trail before a recipient does. Pick a handful of rows at random, including rows from the oldest period and any acquired or migrated source, and walk each pointer to the underlying evidence. If any pointer leads to nothing, or to evidence of a lower tier than the datasheet implies, that is a gaps register entry, not a footnote.

  • Record ID as it exists in the source system, plus the source system and instance name.
  • Created and last modified timestamps from the source, not from the export.
  • Extraction run ID, linking to the query or export job and its date.
  • Transformation run ID and code version for every step that changed the row.
  • Permission reference: the agreement, notice or policy version in force when the record was created.
  • Release ID and the checksum of the file the row was delivered in.
  • Evidence grade for each claim type, so a recipient can filter by strength.

What belongs in an unresolved gaps register?

A gaps register is the list of every claim that falls below the grade the deal needs, with the records it affects and the decision taken about it. It keeps weak claims visible instead of letting them disappear into a summary that reads as fully evidenced. Each entry should carry the fields below; the example column is illustrative and does not describe a real dataset.

What belongs in an unresolved gaps register?
FieldWhat to recordIllustrative entry
Gap ID and claim typeA stable identifier and which of the five claim types is affectedGAP-014, permission
ScopeThe records, period and release the gap applies to, by ID range or filterJob records created before the customer terms update, release r2
Best evidence held and tierThe strongest item available and its current tierDepartment head declaration, tier 5
Evidence that would close itThe specific record or document that would raise the gradeArchived copy of the earlier customer terms with effective date
Consequence while openWhat the dataset documentation must not say until it closesNo statement that pre-update records are covered by customer terms
DispositionClose, narrow scope, exclude records, or disclose as a known limitExclude from release r2; revisit if the archive copy is found
Owner and review dateA named role and the date the entry is next checkedLegal operations lead, next review in 30 days
Change triggerEvents that reopen the entry or require notice to the recipientNew evidence found, records re-included, or terms dispute

Worked example: grading a field service job history

Illustrative scenario. A home services company with a scheduling and dispatch system plans to license six years of job records: work orders, technician notes, parts used and completion codes. Its first datasheet says the records are complete, de-identified and covered by customer terms. Grading splits that into separate claims.

Source grades at tier 1: the dispatch system keeps job status history with timestamps, and the export carries a manifest with a checksum per file. Transformation grades at tier 3 for the current release, because the redaction job ran from a tagged commit with logged row counts, but at tier 4 for an earlier trial export that someone cleaned by hand in a spreadsheet. That trial export is withdrawn rather than defended.

Permission is where the grade drops. Customer terms were updated partway through the period, and the company holds an archived copy of the current terms but only a staff member's memory of the earlier version. Records created after the update grade at tier 2; earlier records grade at tier 6. In this illustration the earlier years are roughly 40 percent of the job records, and they go on the gaps register with a disposition to exclude them from the first release.

The exception is a branch acquired in year four. Its records arrived through a one-time import with no surviving import log, and the only authority evidence is the purchase agreement schedule listing transferred assets. Counsel reads the schedule before anyone grades it, because whether data assets transferred is a legal question; until then, the branch's records stay out of scope and the register notes why.

What the hierarchy cannot settle for you

Strong evidence of a narrow right is still a narrow right. A tier 2 customer agreement proves what it says, and counsel decides whether what it says permits licensing records for model training. The hierarchy tells you how much to trust a document, not how to interpret it.

It also cannot fix missing records after the fact. Reconstruction can raise confidence a little, and independent corroboration can raise it more, but a re-export made today cannot show what a system held three years ago. NIST's generative AI profile suggests audit trails for data flows, including original sources, transformations and decision criteria, which is easier to build forward than to recover backward.

Finally, grades decay. A tier 1 manifest describes one release; a new extraction, a corrected record or a withdrawn permission creates a new state that needs its own evidence. Reopen the register whenever ownership, source systems, permissions or the release itself change.

Bringing graded claims into a licensing review

A graded evidence set shortens rights and data review because every statement arrives with its strongest support and its known limits attached. When operational records are licensed to AI developers through SourceX, it covers sourcing, the rights review, contracting, preparation, delivery and payment, and nothing reaches SourceX until the record owner has approved scope and terms and signed.

The hierarchy and gaps register are worth building before that conversation, though starting it requires no evidence at all: the opening step is a short assessment of the company, with no uploads.

Frequently asked questions

Is a checksum enough to prove where a dataset came from?

No. A checksum proves that a file has not changed since the checksum was recorded, which makes it strong release evidence. It says nothing about which system produced the records, whether they were transformed correctly or whether they may be licensed; those claims need their own evidence.

Why is a signed attestation ranked so low?

An attestation records who stands behind a claim and when, which matters for accountability. It does not add facts that were not already in the underlying records, so its strength depends on the records it summarizes. When those records are strong, cite them directly.

Can reconstructed evidence ever reach the top tier?

Rarely. A reconstruction can be corroborated by independent sources, such as a customer's countersigned order or a processor's statement, which raises confidence for the records it matches. It still cannot show the original system state with the certainty of a record created at the time.

How often should the gaps register be reviewed?

Set a review date for each open entry and reopen the whole register whenever a new release is cut, a source system changes, ownership changes or a permission is withdrawn. Closed entries should stay in the register with the evidence that closed them.

Who should grade dataset provenance evidence?

A data governance lead who understands the source systems should grade source, transformation and release claims, with a pipeline engineer confirming run records. Authority and permission claims need counsel, because grading the document and interpreting what it allows are separate tasks.

Sources

  • Federal Rule of Evidence 803(6) covers records made at or near the time by someone with knowledge as a regular practice, and lets a court exclude them if the circumstances indicate a lack of trustworthiness. Source
  • The committee notes to Rule 902(14) say copied electronic data is ordinarily authenticated by hash value and that the rule establishes authenticity only. Source
  • AS 1105 says reliability depends on the nature and source of evidence and how it was obtained, rates independent outside sources above internal ones, and ranks original documents above copies or digitized versions, whose reliability depends on conversion controls. Source
  • PROV-O models provenance with entities, activities and agents linked by relations such as wasGeneratedBy, wasDerivedFrom and wasAttributedTo. Source
  • The Croissant specification recommends checksums for all files in versioned datasets, no checksums on files still being updated in live datasets, and a version update when past records change. Source
  • SLSA Build L1 provenance can be used to prevent mistakes but is trivial to bypass or forge; at Build L2 a hosted build platform generates and signs the provenance. Source
  • The NIST AI RMF Playbook asks how data provenance is documented, including sources, origins, transformations, augmentations, labels, dependencies, constraints and metadata. Source
  • NIST AI 600-1 suggests audit trails for data and content flows, including original data sources, data transformations and decision-making criteria. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies