Record-Level Provenance: When Dataset-Level Documentation Isn't Enough

Why dataset-level documentation breaks on mixed-rights data

Dataset-level provenance assumes one answer per question, and mixed-rights data has several answers per question. The Data & Trust Alliance provenance standards, a widely covered cross-industry framework, describe whole datasets rather than rows or tables [2][3]. That works for a single licensed corpus. It fails the moment a support-ticket export spans three versions of a customer agreement, or a document set mixes customer-authored files with vendor-authored templates.

The failure mode is quiet. A dataset card says "collected under customer terms permitting service improvement," and that is true for most rows, but some rows came from accounts that signed earlier terms or opted out. Nobody can separate them later because the join key was dropped during preparation. The Data Provenance Initiative reported license omission rates above 70 percent and license error rates above 50 percent on popular dataset hosting sites [4], which shows how quickly coarse metadata drifts from reality.

For a primer on how lineage differs from provenance, see the glossary entry on data lineage. For the category-by-category standard itself, see the Data Provenance Standards explained.

Signals that a dataset needs per-record provenance

Move to record-level tracking when any rights-relevant attribute varies within the dataset and cannot be fixed by splitting it into clean sub-datasets. Five signals come up repeatedly in operational data.

• Consent or legal basis varies. Some rows were collected under explicit consent, others under contract terms, others under a legitimate-interest assessment. Each basis can carry different limits on AI training.
• Notice or terms version varies. Records collected before and after a privacy notice or terms-of-service change may carry different permitted uses. The page on matching each record to the notice in force at collection covers how to reconstruct this.
• Customer or contributor exclusions exist. Enterprise suppliers often hold contractual carve-outs for specific end customers, and individual contributors may have withdrawn. You need a key to drop them now and later, which ties directly to record-level takedown obligations during a license.
• Regulated subsets are mixed in. A claims or support corpus may contain a subset with protected health information that must meet HIPAA de-identification by Safe Harbor or Expert Determination [7] while the rest falls under different rules.
• Downstream use differs by record. Some rows may be cleared for pre-training and supervised fine-tuning but not for retrieval, where text can be surfaced verbatim to end users.

If none of these apply, record-level fields still help debugging, but they are not a rights requirement. Splitting the data into separately documented datasets is often cheaper than building per-row rights machinery.

Record, span and token granularity compared

Granularity should match the smallest unit at which a rights answer can change and at which you might need to delete or explain. Records are the natural unit for structured exports; spans and tokens matter once preparation merges, chunks or packs text.

Illustrative example: invented to show structure; it does not describe an available dataset.

Note that sequence packing for pre-training can place several unrelated records in one training example. If you only stored provenance at the example level, you can no longer say which tokens came from which source, which is the case token-level schemes are designed for [1].

Storing rights compactly: pointers, not repeated text

The efficient pattern is a compact rights identifier on each record that points to a rights table, rather than repeating license text or consent language on every row. A few hundred distinct rights profiles usually cover millions of records, so the per-row overhead is a short string or integer.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "record_id": "tkt-2025-000481",
  "source_system": "helpdesk_export_v3",
  "collected_at": "2025-03-14T09:22:05Z",
  "rights_id": "RP-017",
  "notice_version": "privacy-notice-2024-11",
  "jurisdiction": "US-CA",
  "deid_method": "rule_based_v2",
  "excluded": false,
  "content_sha256": "9f2c...e41a"
}

The rights table then holds one row per profile:

{
  "rights_id": "RP-017",
  "basis": "customer_contract",
  "contract_ref": "msa-2024-template-b",
  "permitted_uses": ["pretraining", "sft"],
  "prohibited_uses": ["rag_verbatim_display"],
  "attribution_required": false,
  "valid_from": "2024-11-01",
  "valid_to": null
}

This design keeps the training shards small and lets counsel change a profile in one place. Field-level definitions for the rights table belong in the permitted-use metadata schema, and dataset-wide summaries can still be expressed in Croissant, the schema.org-based JSON-LD format that describes datasets, file resources and record structure [5]. Keep content_sha256 so you can find a record again after deduplication or reformatting changes its position.

What record-level provenance costs to maintain

Fine-grained provenance is mostly an engineering discipline cost, not a storage cost. The column itself is cheap; keeping it correct through every transformation is where teams fail.

• Every transform must propagate IDs. Deduplication must decide which record_id survives a merge and record the dropped ones. Near-duplicate clustering (MinHash or embedding-based) should log cluster membership so an exclusion applies to all members.
• Chunking and summarization need span maps. A RAG chunk built from three tickets needs three source IDs and offsets, or a takedown cannot find it.
• Synthetic data inherits rights. If a generator was conditioned on licensed seed records, the synthetic rows should carry the seed rights_id values; see provenance records for synthetic training data.
• Validation has to be automated. Add schema checks that reject rows with null rights_id or unknown profiles, for example with JSON Schema validation on dataset deliveries.
• Rights changes must be replayable. When a profile changes, you need a query that lists affected records, shards, chunk stores and training runs.

Budget for this before signing. A dataset that arrives without per-record keys can rarely be retrofitted, because the supplier's source-system IDs are often removed during de-identification.

Access control: fine-grained attribution is itself sensitive

Record- and token-level provenance should be restricted to the people who need it, because the same index that proves a source can be used to single out individual contributors [1]. A table mapping model inputs to record_id, source_system and timestamps can re-link de-identified text to an account, especially when combined with other data.

Practical controls include keeping the rights table and the record-to-source map in a separate store from training shards, issuing pseudonymous record_id values that do not encode customer or user identifiers, logging every attribution query, and limiting queries to aggregate answers (counts by rights_id) for most users. HHS guidance ties Expert Determination to a very small risk that remaining information could identify an individual [7], and a lookup table that reverses pseudonyms raises that risk if it is broadly accessible.

Disclosure, audits and media credentials

Record-level keys make external obligations easier to answer, even where the obligation itself is stated at a higher level. California AB 2013 requires generative AI developers to post documentation about the datasets used to train their systems, with postings due on or before January 1, 2026 [6]; that disclosure is high level, but accurate summaries are much easier to produce from per-record rights data than from memory. Internal and third-party audits follow the same logic, as covered in how to run a provenance audit of an existing training corpus.

For images, audio and video, per-asset provenance may already exist. C2PA Content Credentials attach signed manifests to individual media assets [8], so the asset is the natural record unit; reading C2PA manifests on licensed media explains what to extract and keep.

Questions to put to a supplier before you accept a mixed-rights dataset

Ask suppliers to show, on a sample, that every record can be traced to a rights profile and that exclusions can be applied after delivery. The following checklist can be pasted into a request.

Illustrative example: invented to show structure; it does not describe an available dataset.

1. Which attributes vary across records: consent basis, notice or terms version, jurisdiction, customer exclusions, permitted uses?
2. Is there a stable, pseudonymous record_id that survives de-identification and is not derived from personal data?
3. How many distinct rights profiles exist, and can you provide the profile table with definitions?
4. How were records collected before and after each terms or notice change mapped to versions?
5. Which de-identification method was applied per subset, and was a sample checked?
6. How will you notify us of withdrawals or new exclusions, and at what key?
7. Can you deliver a sample with the record-to-rights mapping so we can test it? See testing a supplier's provenance claims on a sample.

Weak or evasive answers to items 2, 3 and 6 are among the provenance red flags worth escalating before procurement proceeds.

How SourceX handles provenance for sourced datasets

SourceX sources operational datasets, such as support and sales histories, engineering records and documents, from US companies on request, and manages licensing. Each dataset is rights-reviewed for ownership and consents and delivered under a license that defines the records, uses, term and delivery. Personal details are removed or replaced before delivery, the method is recorded, and a sample is checked, though no method is perfect. Diligence materials covering source, rights, preparation and allowed use are prepared per dataset. Buyers who need record-level rights fields can describe that requirement when they submit a data request on the buyers page. The wider cluster lives at the provenance, rights and permitted use hub, within the main AI data guides.

Request mixed-rights data with per-record provenance

SourceX finds US companies that hold the data you describe, then assesses data and licensing permissions before any license is agreed; nothing is contracted until a supplier agrees. Every release is approved by the supplying company, and delivery runs through private, access-controlled workflows after an executed agreement. Describe your dataset and provenance requirements on the SourceX buyers page.