Privacy and preparation
Dataset transformation lineage manifest: from export to delivery
By SourceX Editorial · Updated
9 sections
Short answer
Document dataset transformations with a lineage manifest: a source register of each raw export with its filter, row count and SHA-256 hash, then one row per transformation step run with input and output hashes, code commit, parameters, row counts, drop reasons, operator and checks. Give every delivered row a lineage key back to its source, version each release, and validate the chain before delivery.
Provenance trail
Dataset transformation
Key takeaways
- A lineage manifest records source exports, every transformation step run and the release, linked by content hashes.
- Each step needs its code commit, parameter hash, row counts and a reason code for every dropped row, or it cannot be reproduced.
- A keyed lineage key on every delivered row lets a reviewer trace it to its source and permission without exposing the original ID.
- A validator should check the hash chain, row reconciliation, reproducibility, permission coverage, record trail and version rule.
- Editing records after release requires a new MAJOR version and a change notice, never an in-place overwrite.
How do you document dataset transformations from source export to delivery?
Document dataset transformations with a lineage manifest that lists every source export, every transformation step in order, and the release that came out the other end. Each step records its inputs and outputs by content hash, the exact code version and parameters, row counts in and out with reasons for every dropped row, who ran it, and which permission covered it. A reviewer should be able to start from any delivered record and walk back to the source export without asking anyone.
The test of a good manifest is reproducibility by a stranger. If an auditor, a buyer's data team or your own successor can rerun the steps from the recorded code and parameters and get files with the same hashes, the manifest works. If they need a meeting to understand what happened between the export and the delivery, it does not.
Start the manifest before the first export, not after delivery. Lineage reconstructed from memory and old notebooks is always incomplete, while lineage written as each step runs costs a few minutes per step.
The worked example on this page is illustrative. The system, file names, counts, dates and identifiers are invented to show structure and do not describe any company's data.
What a transformation lineage manifest is, and what it is not
A transformation lineage manifest is the step-by-step record of how delivered files were derived from source exports. It differs from a scope manifest, which lists what tables and fields a dataset contains, and from a data dictionary, which defines each field. You need all three, but only the lineage manifest answers what was done to the data and in what order.
The vocabulary already exists. The W3C PROV Ontology describes provenance with entities, activities and agents: an entity can be derived from another entity, an activity uses and generates entities, and an agent is associated with an activity or has an entity attributed to it. OpenLineage, an open specification for lineage metadata, models jobs, runs and datasets, requires every run to carry a UUID, and offers facets for source code location with a git SHA, dataset versions, output row counts and column-level lineage. The manifest below maps directly onto both, so it can be exported to either format later.
Dataset documentation frameworks ask for the same evidence in prose. Datasheets for Datasets proposes that every dataset be accompanied by documentation of its motivation, composition, collection process and recommended uses, and the Croissant responsible AI vocabulary includes fields for data preprocessing and data manipulation protocols. A lineage manifest is the structured record from which those prose sections can be written accurately.
The manifest header and source register
The header identifies the release and the permission behind it; the source register lists each raw export exactly as it left the source system. Without a source register, every later hash proves only that files were consistent with each other, not that they came from where the manifest says.
Record each export at the moment it was taken, with its filter or query text, the period it covers, the row count the source system reported, and a SHA-256 hash of each file. FIPS 180-4, the Secure Hash Standard, defines SHA-256 and states that hash algorithms are used to detect whether messages have been changed since the digests were generated. A hash does not prove who produced a file; pair it with the access log or an export receipt from the source system.
Note the source system's own limits next to each export. Many business applications cap export sizes, page results, truncate long text fields or export only current values rather than history. If the export tool returned pages that overlapped, or a field was cut at a fixed length, that belongs in the source register, because every later step inherits it and a reviewer will otherwise blame a transformation for a gap that was there from the start.
| Field | Purpose | Illustrative value | Validation rule |
|---|---|---|---|
| manifest_id, release_version | Identify this manifest and the release it describes | LM-ILL-0007, 1.0.0 | Version follows MAJOR.MINOR.PATCH |
| permission_ref | Point to the approved scope and agreement covering the data | SCOPE-ILL-12, signed version 2 | Must resolve to a signed record dated before the first step |
| source_id, system, export_method | Name each raw export and how it was taken | SRC-1, help desk tickets, admin CSV export | One entry per export run |
| export_filter, period | Show exactly what was selected | Closed tickets, Jan 2023 to Dec 2025 | Filter text stored verbatim |
| files, sha256, row_count | Fix the content of each raw file | tickets.csv, comments.csv with hashes and counts | Hashes recompute; counts match source system report |
| exported_by_role, exported_at | Attribute the export | Support operations admin, 2026-01-14 | Role exists in the access log for that date |
The transformation step record: one row per step run
Every transformation step gets one row each time it runs, with enough detail to rerun it. A step that cannot be rerun from its recorded code and parameters is a gap in the lineage, even if its output looks correct.
The fields below are the minimum. The two that teams most often skip are the parameter hash and the drop reasons. Parameters such as a date cutoff or a redaction pattern list change outputs as much as code does, and a row count that falls from 412,000 to 388,000 means nothing to a reviewer until each removed row has a reason code.
Steps that are not deterministic need extra fields. A random sample needs its seed. A model-assisted redaction step needs the model name and version, the prompt or rule file, and a record of which outputs a person reviewed. A manual correction made in a spreadsheet needs to become a step of its own, with the edited file hashed before and after and the editor's role recorded, because a manifest that hides hand edits inside an automated step misstates how the data was made.
| Field | What it records | Illustrative value | Why a reviewer needs it |
|---|---|---|---|
| step_id, order, run_id | Which step, its position, which execution | S3, 3, UUID per run | Distinguishes reruns of the same step |
| step_type | Filter, join, de-identify, deduplicate, sample, split or format | De-identify | Tells the reader what kind of change to expect |
| inputs, outputs | Artifact IDs with SHA-256 hashes | A2 in, A3 out | Chains steps together without ambiguity |
| code_ref, params_hash | Repository commit and hash of the configuration file | Commit 9f1c2e7, params hash recorded | Makes the step reproducible |
| rows_in, rows_out, drops_by_reason | Counts and reason codes for removed or added rows | 388,240 in, 388,240 out, 0 dropped | Proves nothing was lost silently |
| fields_changed | Columns created, altered or removed | requester_name, requester_email, body | Feeds column-level lineage |
| operator_role, started_at, ended_at | Who ran it and when | Data engineer, 2026-01-16 | Attribution and ordering |
| checks | Tests run on the output and their result | PII scan on body: pass | Shows the step did what it claimed |
Worked example: an illustrative help desk export, step by step
The illustrative chain below takes two raw help desk exports to a delivered JSONL release in six steps. Read it as a pattern: each output artifact becomes the next step's input, every count reconciles, and every row removed has a reason.
Note what the chain does not do. It does not rename a field without recording it, it does not drop rows inside the de-identification step, and it does not reshuffle the split after checks pass. Each of those habits makes lineage impossible to follow because one step quietly does two jobs.
Two details in the chain are worth copying. First, S2 cites the exact scope record that excluded certain customer accounts, so a reviewer can see that 9,272 tickets left for a permission reason rather than a quality reason. Second, S5 keeps tickets with no comments instead of dropping them in the join, and says so; an inner join that silently removed them would shrink the dataset and skew it toward longer conversations without any line in the manifest to explain why.
| Step | Type and action | Input to output | Rows in, out | Drop reasons recorded |
|---|---|---|---|---|
| S1 | Filter: closed tickets in the approved period | A0 (SRC-1) to A1 | 412,906 to 397,512 | 15,394 outside period or not closed |
| S2 | Filter: remove tickets outside approved scope | A1 to A2 | 397,512 to 388,240 | 9,272 from customer accounts excluded in SCOPE-ILL-12 |
| S3 | De-identify: pseudonymize people, redact free-text identifiers | A2 to A3 | 388,240 to 388,240 | None; field changes listed |
| S4 | Deduplicate: merge tickets with identical source key | A3 to A4 | 388,240 to 388,101 | 139 duplicates from overlapping export pages |
| S5 | Join: attach comments (SRC-2) by ticket key | A4 plus comments to A5 | 388,101 to 388,101 | None; 2,310 tickets had no comments, kept |
| S6 | Format and split: JSONL, about 90/10 by ticket creation month | A5 to R1-train, R1-test | 388,101 to 349,317 plus 38,784 | None; split rule recorded |
The record-level trail: tracing one delivered row back to its source
Record-level lineage means every delivered row carries a pointer back to its source record and to the release that contains it, without exposing the original identifier. A keyed hash of the source system's record key works well: it is stable across steps, it lets the data owner look up the original, and it does not reveal the key to anyone without the secret.
At a glance
The record-level trail: tracing one delivered row back to its source
With that pointer and the step table, a reviewer can answer the question every buyer eventually asks about a specific row: where did this come from, what was changed, and under which permission. For the illustrative row, the answer is source export SRC-1, filtered by S1 and S2, pseudonymized in S3, kept as the surviving copy in S4, joined with comments in S5, and placed in the training split by S6, all under SCOPE-ILL-12.
Joins and merges need one more rule. When S4 merges duplicates, record which source key survived and which were folded into it; when S5 attaches comments, each comment keeps its own lineage key from SRC-2. A delivered row can then have several ancestors, and the trail lists all of them rather than only the parent ticket.
- lineage_key: keyed hash of the source ticket ID, identical in every artifact from A0 to R1 (illustrative).
- source_id: SRC-1 for the ticket, SRC-2 for each joined comment.
- steps_applied: S1, S2, S3, S4, S5, S6, in order, each pointing to its run_id.
- fields_transformed: requester_name and requester_email replaced in S3; body redacted in S3; comments added in S5.
- permission_ref: SCOPE-ILL-12, inherited from the source register entry.
- release_id and file: 1.0.0, R1-train part 004, with the file's SHA-256 hash.
Validator: the checks a manifest must pass before release
A manifest is only trustworthy if a script can check it. Run the validator below on every manifest before a release is sealed, and store the validator's output next to the manifest so the check itself is part of the evidence.
Run it twice: once by the team that built the pipeline, and once by someone who did not, starting from the manifest alone. The second run is the real test, because the builder fills gaps from memory without noticing. For the record-trail check, draw the sample across every source, step outcome and split rather than from the first file, and record the sample size and the rows drawn so the check can be repeated.
| Check | Rule | Passes when | Typical failure it catches |
|---|---|---|---|
| Hash chain | Every input hash equals an earlier output or source hash | No orphan inputs | A file edited by hand between steps |
| Hash recompute | Recompute SHA-256 for every listed file | All match | A release file replaced after sealing |
| Row reconciliation | rows_out equals rows_in minus drops plus documented additions | Every step balances | Silent row loss in a join or filter |
| Reproducibility | Each step has code_ref and params_hash | No step missing either | A notebook cell run once and never saved |
| Permission coverage | Every source has a signed permission_ref dated before step S1 | All sources covered | Data processed before scope approval |
| Record trail | Sample of delivered rows resolves back to a source record | 100 percent of the sample resolves | Lineage key regenerated mid-pipeline |
| Version rule | Release version change matches the kind of change | Bump is MAJOR, MINOR or PATCH as required | Edited records shipped under the old version |
Failure example: a quiet rerun after release
The most common lineage failure is a step rerun after release without a new version. It looks harmless because the files are still in the same place with the same names.
In an illustrative failure, a reviewer finds a missed phone number pattern after release 1.0.0 ships. An engineer updates the redaction list, reruns S3 through S6 and overwrites the release files in place. The manifest still says 1.0.0 with the old hashes and the old S3 params_hash. The validator fails three checks at once: the recomputed hashes of the release files no longer match the manifest, the S3 run has no row, and the version rule is broken because existing records were edited.
Semantic Versioning 2.0.0 states that once a versioned package has been released, its contents must not be modified, and any modifications must be released as a new version. The correct fix is a new S3 run with its own run_id, new downstream runs, and a new release with its own hashes, plus a change notice telling every recipient of 1.0.0 what changed and why.
Versioning, change notices and who signs off
Version numbers should tell a recipient what kind of change happened without reading the diff. The Croissant 1.1 specification recommends MAJOR.MINOR.PATCH versioning for datasets following Semantic Versioning: PATCH when the data is the same but serialized or packaged differently, MINOR when existing data is unchanged and data has been added, and MAJOR when existing data has been edited, removed or moved across splits. It also strongly recommends recording SHA-256 checksums for files in versioned datasets.
Under that rule, the redaction fix above is a MAJOR change, because existing records were edited. Adding a later quarter of tickets in a separate split is MINOR. Converting the same records from JSONL to Parquet is PATCH. Each release gets a short change notice listing the steps rerun, the reason, the row-count effect and the hashes of the new files.
Sign-off belongs in the manifest, not in an email thread. Record who approved each release by role, which checks passed, and which permission it relies on. Keep access permission, privacy permission, ownership and contractual licensing rights as separate references: a manifest proves what was done, and de-identification recorded in it does not by itself create a right to license the result.
When a dataset is licensed through SourceX, the company approves the scope and terms and signs the agreement before any data is delivered, and de-identification is performed by SourceX through approved processing routes. A manifest like this one is how either side can show, record by record, that the delivered files match what was approved.
Frequently asked questions
What should the transformation lineage and version manifest include?
It should include a header with release version and permission reference, a source register with each raw export's filter, period, row count and SHA-256 hash, and one row per transformation step run with inputs, outputs, code commit, parameter hash, row counts, drop reasons, operator and checks. It should also define a record-level lineage key and a change notice for each release.
Can a reviewer trace a claim to the exact record and release evidence?
Yes, if every delivered row carries a lineage key and a release ID, and the step table records which runs touched it. The reviewer follows the key from the release file back through each step's artifacts to the source export entry and its permission reference. A sample of delivered rows should resolve completely before release.
What valid and invalid examples demonstrate the declared structure?
A valid manifest has a continuous hash chain, balanced row counts, code and parameter references for every step, and a version bump that matches the change. The invalid example on this page is a quiet rerun after release: the files changed, the hashes and version did not, and the validator fails three checks.
What is the difference between data lineage and data provenance?
Lineage usually describes how data moved and changed through systems and steps. Provenance is broader and includes origin, responsibility and the rights or permissions attached to data. A transformation lineage manifest records both: the steps, and the agents and permission references behind them.
Do I need OpenLineage or PROV tooling to keep a lineage manifest?
No. A table or JSON file with the fields on this page is enough to start. The fields map onto OpenLineage runs, jobs and datasets and onto PROV entities, activities and agents, so the manifest can be exported to either format if a recipient asks for it.
Is a checksum enough to prove a file was not tampered with?
A SHA-256 checksum detects whether a file changed since the hash was computed. It does not prove who produced the file, and someone who can replace both the file and the recorded hash can defeat it. For delivery, sign the manifest or exchange hashes through a separate channel.
Sources
- The W3C PROV Ontology describes provenance with entities, activities and agents, linking derived entities with wasDerivedFrom and attributing responsibility with wasAssociatedWith and wasAttributedTo. Source
- The OpenLineage specification models jobs, runs and datasets, requires each run ID to be a UUID, and defines facets for source code location with git SHA, dataset version, output statistics and column lineage. Source
- The Croissant 1.1 specification recommends MAJOR.MINOR.PATCH dataset versions with defined meanings for each bump and strongly recommends SHA-256 checksums for files in versioned datasets. Source
- The Croissant responsible AI vocabulary defines fields including rai:dataPreprocessingProtocol and rai:dataManipulationProtocol for describing how data was processed. Source
- Semantic Versioning 2.0.0 states that once a versioned package is released its contents must not be modified and any modifications must be released as a new version. Source
- Datasheets for Datasets proposes that every dataset be accompanied by documentation of its motivation, composition, collection process and recommended uses. Source
- FIPS 180-4, the Secure Hash Standard, specifies SHA-256 and states that hash algorithms are used to detect whether messages have been changed since the digests were generated. Source
See if your company qualifies
A short company assessment. No data uploads are needed.