Privacy and preparation
Dataset metadata evidence links: tying each dataset claim to its proof
By SourceX Editorial · Updated
10 sections
Short answer
To link every material dataset claim to its evidence, give each claim in the data card an ID and record, in a claim to evidence index, the artifact that proves it, a locator pinned to the release (version, checksum or run ID), how strong that evidence is, what it does not show and what would make it stale. Claims that cannot be traced are narrowed, dropped or escalated.
Retrieval
Dataset metadata evidence
Key takeaways
- Index claims, not paragraphs: one row per material sentence per release.
- A locator must identify the evidence as it was, using versions, checksums or pipeline run IDs rather than live paths.
- Match evidence strength to the cost of being wrong; removal and rights claims need more than one kind of proof.
- Record what the evidence does not show, so the published sentence is never broader than its support.
- Test the index by having someone who did not build the dataset trace random claims without help.
- Narrow or drop untraceable claims, and send rights or personal data gaps to counsel or a privacy reviewer.
How do you link every material dataset claim to its supporting evidence?
Give each material claim in the dataset documentation an ID, then record in a claim to evidence index the exact artifact that proves it, a locator that will still resolve after the next release, the strength of that evidence and the event that would make it stale. A claim with no row in the index is either rewritten until it can be proven or taken out of the card.
Most dataset documentation is written as prose. A datasheet or data card says the records cover three years, that customer names were removed, that the supplier holds the right to license them. Each of those sentences is a promise, and a buyer, an auditor or the supplier's own counsel will eventually ask what stands behind it. If the answer lives in someone's memory, the claim is only as good as that person's availability.
The index changes the unit of documentation from the paragraph to the claim. It does not replace a datasheet or data card; it sits behind one, so that every sentence a reader might rely on has a pointer to a file, a query, a pipeline run or a signed document, pinned to the release the card describes.
Which sentences in a data card are material claims and which are context?
A claim is material when a reader would make a different decision if it were false: whether to license, how to use the data, what to disclose downstream, or whether a model trained on it can ship. Background prose about why the dataset exists is context, not a claim, and does not need an evidence row.
At a glance
Which sentences in a data card are material claims and which are context
Documentation frameworks already ask the questions that produce claims. Datasheets for Datasets proposes that every dataset come with a datasheet covering its motivation, composition, collection process and recommended uses. The Croissant responsible AI vocabulary adds machine-readable properties such as rai:dataCollection, rai:dataPreprocessingProtocol, rai:personalSensitiveInformation, rai:dataLimitations and rai:dataReleaseMaintenancePlan. Hugging Face dataset cards carry YAML metadata such as license, language and size above the prose. Every answer written into one of those fields is a candidate claim.
- Source claims: which system the records came from, which accounts or sites, which business entity held them.
- Coverage claims: date range, record counts, languages, regions, share of records with a given field populated.
- Transformation claims: what was removed, masked, normalized, deduplicated or relabeled, and by which pipeline version.
- Permission claims: the contract, notice or authority that allows the records to be licensed for this use.
- Exclusion claims: opt-outs honored, categories filtered out, customers or matters withheld.
- Quality claims: label agreement, error rates found in review, known gaps.
- Release claims: version, file checksums, what changed from the previous release.
The claim to evidence index, field by field
Keep one row per claim per release. The fields below are enough to answer the question a reviewer actually asks, which is not whether evidence exists somewhere but whether this sentence, in this version of the card, can be checked against something that has not moved. The examples are illustrative.
Two fields do most of the work. The locator decides whether the evidence can be found again, and the limits field decides whether the published sentence is honest about what was proven. Teams that skip the limits field tend to publish claims that are true for the part of the data they tested and silently stretched to the rest.
| Field | What to record | Illustrative entry | Weak when |
|---|---|---|---|
| Claim ID and text | A stable ID and the exact sentence as published in the card | C-07: Customer email addresses were removed from ticket bodies | The text in the index drifts from the text in the card |
| Claim type and scope | Source, coverage, transformation, permission, exclusion, quality or release; and whether it applies to the whole dataset, a subset or each record | Transformation; every record in the ticket_body field | Scope is left implicit, so a subset claim reads as universal |
| Evidence artifact | The specific file, query, run log, report or signed document | Redaction run log and the post-run pattern scan report | It names a folder or a team rather than an artifact |
| Evidence locator | A path plus version, hash, run ID or document date that will still resolve later | Run ID with the output file checksum for release 2.1.0 | It points to a live location that will be overwritten |
| Record-level link | How a reviewer gets from the claim to sample records that show it | Record IDs sampled in the scan, stored with the report | Only aggregate figures exist |
| Evidence strength | Direct, run-logged, sampled, attested or inferred | Run-logged plus sampled | An attestation is recorded as if it were a test |
| Owner and verifier | Who maintains the evidence and who last checked the link | Data engineering lead; privacy reviewer | One person both writes and checks |
| Last verified | Date and release the link was last opened and checked | 2026-10-18 against 2.1.0 | The date predates the release the card describes |
| Invalidation trigger | The event that makes this row stale | Any change to the redaction rules or a new source system | No trigger, so the row is never revisited |
| Limits | What the evidence does not show | Scan covers ticket bodies, not attachments | Left blank, so the claim reads as broader than its proof |
What makes an evidence locator stable enough to cite?
A locator is stable when it identifies the evidence as it was when the claim was made, not as it is today. File paths and dashboard links fail this test because the thing at the end of them changes. Versions, content hashes and run identifiers pass it.
Croissant gives two of these for free. Its specification recommends versioning datasets with MAJOR.MINOR.PATCH, with a major increment when existing records are edited, removed or moved between splits, and it lets each file object carry a sha256 checksum so a downloaded file can be matched to the one the metadata describes. A claim that cites release 2.1.0 and the checksum of the file it was tested on is anchored; a claim that cites the dataset name is not.
Transformation claims need the pipeline run, not just the code. In OpenLineage, a run is an instance of a job with its own globally unique ID, which the specification requires to be a UUID, and each run event names the input and output datasets. A DatasetVersion facet can record the version of a dataset a run read or wrote, and the column lineage facet maps each output field to the input fields used to compute it, with a flag for whether the transformation masked the data. Citing that run ID ties a sentence such as 'names were masked' to the execution that did it.
When the evidence is a relationship rather than a file, the W3C PROV ontology gives the vocabulary: an entity wasGeneratedBy an activity, wasDerivedFrom another entity, and wasAttributedTo an agent. Croissant 1.1 shows how to attach these properties at the dataset and field level, which is useful when a field was derived from an upstream system that has its own evidence trail.
Contracts and notices need a locator too. Record the document version or effective date and where the signed copy is held, not just the contract name. A permission claim that cites a customer agreement without saying which version was in force for which records is the kind of row that fails review, because terms change over the years a dataset covers.
How strong does the evidence behind each claim type need to be?
Match the strength of the evidence to the cost of the claim being wrong. A record count can rest on a query log. A statement that personal data was removed or that the supplier may license the records needs more than one kind of evidence, because a single artifact can be incomplete in ways it cannot reveal by itself.
| Strength | What it means | Enough on its own for | Needs a second form for |
|---|---|---|---|
| Direct | The claim is visible in the delivered records themselves | Field presence, value formats, record counts | Nothing further when the check is reproducible |
| Run-logged | A logged pipeline run with inputs, outputs and versions produced the result | Deduplication, normalization, field derivation | Masking and removal claims, which also need a sample check |
| Sampled | Reviewers inspected a documented sample and recorded findings | Label quality, residual error rates | Universal claims such as none or all, which a sample cannot prove |
| Attested | A responsible person signed a statement | Facts only that person can know, such as contract history | Anything that can also be checked in records or logs |
| Inferred | The claim follows from other evidence by reasoning | Low-stakes descriptive context | Every material claim; reword it or find direct evidence |
Worked example (illustrative): indexing a support ticket data card
Take a hypothetical field service company preparing a support ticket dataset from its help desk. Its draft data card makes five material claims. The table shows how each was indexed. Counts and dates are illustrative.
One claim did not survive. The draft said the dataset was representative of all customers. The only evidence was a distribution of tickets by product line, which says nothing about customers who never opened a ticket. Rather than invent support for it, the team narrowed the sentence to describe ticket distribution by product line and recorded the rest as a limit.
The permission row is the one to read closely. The team could not find signed copies of two older contract templates, so it did not claim that every ticket was covered. It withheld the tickets created under those templates and said so in the limits field. That exception cost some volume and kept the claim true.
| Claim as published | Evidence artifact and locator | Strength | Limit recorded |
|---|---|---|---|
| Tickets opened between January 2023 and June 2026 | Export query text and its result count, pinned to release 1.0.0 | Direct | Reopened tickets keep their original open date |
| Customer email addresses removed from ticket bodies | Redaction run ID, output checksum, scan of 400 sampled records with record IDs | Run-logged plus sampled | Attachments were excluded, not scanned |
| Tickets from customers who opted out are excluded | Opt-out list snapshot dated before the export, join log showing excluded IDs | Run-logged | Opt-outs received after the snapshot date are handled in the next release |
| Company may license these records for AI training | Signed customer terms versions in force during the period, counsel's memo reference | Attested plus document review | Two legacy contract templates under review; their tickets withheld |
| Resolution codes reviewed by a second agent | Review log with reviewer IDs and disagreement rate on a sample | Sampled | Review began in 2024; earlier tickets carry single-agent codes |
Can a reviewer trace a claim to the exact record and release?
Run a trace test before any card is published: hand a reviewer who did not build the dataset three claims chosen at random and ask them to reach the evidence using only the index. If they need to ask someone where a file is, the locator is not good enough.
At a glance
Can a reviewer trace a claim to the exact record and release
The test also catches stale rows. Each row records the release it was last verified against, so a reviewer can see at once when a card describes release 2.1.0 but a claim was last checked on 2.0.0. The invalidation trigger tells the dataset owner which rows to reopen when a source system, a redaction rule or a contract changes; Croissant's rai:dataReleaseMaintenancePlan property is a natural place to state that this review happens on each release.
A failed trace is useful information, not an embarrassment. It shows which kinds of evidence the team stores well and which it reconstructs from memory. In most operational datasets the weak spots are the same: contract versions, the date an exclusion list was taken, and what happened to attachments and free-text fields that a redaction step was never pointed at. Fixing those once makes every later release cheaper to document.
- Pick claims across types, including at least one permission or transformation claim.
- From the claim ID, open the evidence artifact by its locator, without searching.
- Confirm the artifact's version, hash or run ID matches the release named in the card.
- Follow the record-level link to sample records and confirm they show what the claim says.
- Read the limits field and check that the published sentence is no broader than the evidence.
- Record the result, the date and the reviewer in the row.
Where should the index live, and who keeps it current?
Keep the index next to the dataset release, versioned with it, so that release 2.1.0 of the data ships with release 2.1.0 of the index. A spreadsheet in a shared drive drifts from the data within one release cycle; a file committed alongside the dataset metadata does not.
Split ownership by evidence type. Data engineering owns rows backed by pipeline runs and checksums, the privacy reviewer owns removal and masking rows, and whoever manages customer contracts owns permission rows. A separate verifier signs off each row, so the person who produced the evidence is never the only one who checked the link.
Machine-readable metadata helps but does not remove the need for the index. Croissant and OpenLineage carry versions, checksums, runs and lineage in structured form, which makes many locators cheap to record. They do not say which sentence in the card a run supports, how strong that support is or what it leaves out. The index is where those judgments are written down.
When a claim cannot be traced: narrow it, drop it or escalate it
If the evidence covers part of the claim, narrow the sentence to that part and put the remainder in the limits field. A coverage claim that holds for one region becomes a claim about that region. This is usually the right answer and costs nothing but precision.
If no evidence exists and none can be produced, drop the claim. Leaving an unsupported sentence in a card is worse than leaving a gap, because a reader will treat the card as checked.
Treat absolute words as a warning sign. Claims that say all, none, every or fully need evidence that covers the whole population, and a sample cannot provide it. Rewrite them to state what was done and what was checked, such as a rule applied to every record and a sample reviewed afterward, unless the evidence really is exhaustive.
Escalate when the missing evidence concerns rights or personal data. A permission claim that rests on contracts nobody has located, or a de-identification claim with no run log, is not an editing problem; it goes to counsel or a privacy reviewer before the dataset moves, and the affected records are held back in the meantime.
Bringing an indexed dataset to a licensing review
A claim to evidence index shortens any licensing review because the reviewer's questions already have pointers. SourceX's work for companies licensing operational records directly to SourceX includes rights review, contracting, preparation and delivery, and an index is the material that review reads first. Since the company approves the scope and terms and signs before delivery, an evidence gap found in review can be closed by narrowing scope instead of rewording the card. Qualification starts with a short assessment, with no data uploaded.
Frequently asked questions
What is a claim to evidence index for a dataset?
It is a table behind a datasheet or data card with one row per material claim. Each row names the artifact that proves the claim, a locator pinned to a release, the strength of the evidence, its limits and the event that would make it stale.
Is a data card the same as dataset provenance?
No. A data card describes the dataset in prose and structured fields. Provenance records where the data came from and what was done to it. The index connects the two, so each statement in the card points to the provenance or contract evidence behind it.
How do I cite a pipeline step as evidence?
Cite the specific run, not the code repository. Record the run ID, the input and output dataset versions and the checksum of the output file, so a reviewer can confirm the step that produced the released data is the one described.
Can a signed attestation support a data rights claim by itself?
It can support facts only the signer can know, such as contract history. Where the claim can also be checked in documents, such as the customer terms in force during the period, record that document review as well.
How often should evidence links be rechecked?
On every release, and whenever a row's invalidation trigger fires, such as a change to a source system, a redaction rule or a contract template. Each row should show the release it was last verified against.
Sources
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- The Croissant RAI vocabulary defines properties including rai:dataCollection, rai:dataPreprocessingProtocol, rai:personalSensitiveInformation, rai:dataLimitations and rai:dataReleaseMaintenancePlan. Source
- Hugging Face dataset cards are README files with a YAML metadata section for fields such as license, language and size. Source
- Croissant recommends semantic versioning of datasets, sha256 checksums for file objects, and shows dataset- and field-level provenance with PROV properties. Source
- In OpenLineage a run is an instance of a job identified by a UUID, and run events name the job's input and output datasets. Source
- The OpenLineage DatasetVersion facet records the version of a dataset. Source
- The OpenLineage column lineage facet maps output fields to the input fields used to compute them, with transformation type and a masking flag. Source
- PROV-O relates entities, activities and agents with properties including wasGeneratedBy, wasDerivedFrom and wasAttributedTo. Source
See if your company qualifies
A short company assessment. No data uploads are needed.