Skip to content

Systems and records

Dataset version identifiers: give each release an identity you can reconcile

By SourceX Editorial · Updated

9 sections

Short answer

Give each dataset release two identifiers: a readable version label and a release digest, the SHA-256 of a canonical manifest that lists every file hash, the record ledger hash, transformation versions and permission references. The label points to the digest and is never reused. Because the digest covers record-level evidence, anyone can later recompute it and trace a disputed record to its exact release.

System export

Dataset version identifiers

This page covers dataset version identifiers: give each release an identity you can reconcile.Illustrative

Key takeaways

  • A version label is a claim; the release digest, computed from a canonical manifest, is the identity.
  • Hash a manifest that covers files, the record ledger, transformation versions and permission references, not just the files.
  • Canonicalize JSON with RFC 8785 and prefix every hash with its algorithm so independent tools agree.
  • A one-row-per-record ledger with keyed source references turns a release hash into record-level answers.
  • Never reissue a label with new bytes; ship a new version with the old digest as parent.

How do you assign dataset version identities that support later reconciliation?

Give every release two identifiers and keep them apart: a human label such as [email protected] that people say out loud, and a release digest computed from a canonical manifest that machines compare. The label points at the digest; the digest is the identity. A release is the same release only when the digest matches, whatever anyone calls it.

The manifest behind the digest has to carry more than file hashes. To reconcile later you need to answer which records went out, from which source extract, after which transformation, under which permission, and to which recipient. So the manifest also points to a record ledger with one row per record, and that ledger has its own hash inside the manifest. Change any record, any redaction pass or any permission reference and the digest changes with it.

This page gives a release ID, checksum and manifest convention a data owner can adopt as is, a validator checklist, and one valid and one failing example. The goal is narrow: when a buyer, an auditor or your own counsel asks about a specific record in a specific delivery, you can answer from evidence rather than memory.

Why a version label cannot settle a question asked two years later

A label is a name, and names get reused. Someone fixes a bad export on a Friday, regenerates the files and ships them again as 2.1.0 because the change felt small. Six months later two parties hold different bytes under the same label, and neither can prove which one the other has. Semantic Versioning forbids exactly this: once a versioned package is released, its contents must not be modified, and any change must be released as a new version. The rule only helps if something mechanical enforces it, and a digest does.

Labels also mean different things to different conventions. Under Semantic Versioning a patch is a backward compatible fix. The Croissant metadata specification, in both version 1.0 and the current 1.1, recommends MAJOR.MINOR.PATCH for datasets but defines a patch as data that stays the same while its serialization or packaging changes, and a major as existing records edited, removed or shuffled across splits. A corrected record is a patch under one reading and a major under the other. Pick one, write it in the manifest as a version policy, and never let the label carry the identity.

Storage-level identifiers do not solve this on their own. Apache Iceberg tracks snapshots with unique snapshot IDs and lets you place a tag, a label for one snapshot, or a branch, a mutable reference that moves when new snapshots are committed. Its retention rules can remove references and expire old snapshots. A snapshot ID is excellent inside the table and meaningless to a buyer who received exported files, so record it in the manifest as a source pointer, not as the release identity.

Which identifiers does each release carry, and what does each one prove?

Each release needs five identifiers, and each answers a different question. Keeping them separate is what lets a reviewer check one claim without trusting the others.

  • Build metadata is a safe place for a short digest prefix. Semantic Versioning ignores build metadata when ordering versions, so 2.1.0+m.4be1c09a7d2f still sorts as 2.1.0 while letting people spot a mismatch at a glance.
  • Never use the short prefix as the authority. It is for eyes; comparisons use the full digest.
Which identifiers does each release carry, and what does each one prove?
IdentifierForm (illustrative)What it provesWho can recompute it
Dataset IDhelpdesk-ticketsWhich line of releases this belongs to; never changes across versionsNobody needs to; it is assigned once and registered
Version label2.1.0, with optional build metadata such as 2.1.0+m.4be1c09a7d2fWhere the release sits in the sequence and what kind of change the policy says it isNobody; it is a claim, checked against the digest
Release digestsha256: followed by 64 hex characters, over the canonical manifestThat two parties hold the identical manifest, and through it identical files and ledgerAnyone holding the manifest
Parent digestDigest of the previous release, or none for the firstThe release history cannot be reordered or silently forkedAnyone holding both manifests
Record content hashsha256 over one record's canonical delivered formThat a disputed record is byte-identical to what was deliveredAnyone holding the record and the ledger row

How do you make the release digest come out the same on every machine?

Hash a canonical serialization of the manifest, not whatever file your script happened to write. Two exports of the same manifest can differ in key order, whitespace or number formatting, and each difference produces a new digest for identical content. That turns every comparison into a false alarm, and people learn to ignore alarms.

The JSON Canonicalization Scheme, published as RFC 8785, exists for this. It produces a hashable representation of JSON by sorting object properties at every level by their UTF-16 code units, leaving array order untouched, and serializing primitives with strict ECMAScript rules. If your manifest is JSON, canonicalize with JCS, hash the resulting bytes with SHA-256, and write the result with an algorithm prefix.

Borrow the prefix grammar from the Open Container Initiative image specification, which treats a digest as a content identifier: an algorithm, a colon, then the encoded hash, as in sha256 followed by 64 hex characters. The specification recommends SHA-256 and describes verification as recalculating the digest independently and comparing. The Frictionless Data Resource specification makes the same move at file level: its hash property defaults to MD5 but other algorithms are indicated by prefixing the value with the algorithm name. Always write the prefix so nobody has to guess the algorithm from the length.

Decide what the digest excludes, and say so in the manifest. Exclude anything added after the release is sealed: signatures over the digest, transfer receipts, recipient lists. Include everything that defines content: files, ledger hash, transformation versions, permission references and the parent digest. Arrays whose order does not matter, such as a file list, should be sorted by path before hashing, because JCS deliberately keeps array order as given.

The release manifest convention, field by field

The manifest is the asset this page exists to give you. Each field below is required unless marked optional, and every field has one job. Values in the second column are illustrative.

At a glance

The release manifest convention, field by field

The release manifest convention, field by field
FieldExample value (illustrative)PurposeValidator rule
dataset_id and versionhelpdesk-tickets, 2.1.0Human handle for the release line and positionPair must map to exactly one release digest across all published manifests
version_policysemver-2.0.0-data-v1, a link to your written rulesStates what a patch, minor and major mean for this datasetMust resolve to a published policy document
parent_releasesha256 digest of 2.0.3Chains releases; exposes forksMust equal the digest of an existing manifest, or be null for the first release
source_registerSystem name, export job ID, extraction window start and end, source snapshot ID if anyWhere every record came from and when it was readEvery ledger row's source reference must fall inside a listed extract
transformationsOrdered steps with tool name, tool version and a config digestWhich de-identification, filtering and normalization ranEach step needs a version and a config digest; no free-text steps
permission_refsAgreement ID, scope schedule ID, any data-subject notice or consent basis IDThe documents that allowed this release to existEvery ID must resolve in the permission registry and be in force on the release date
filesSorted list of path, bytes, sha256, record_countByte-level completeness and integrityRecomputed hashes and sizes must match; no unlisted files
ledgerPath and sha256 of the record ledger, plus row countBinds the record-level trail into the release identityLedger hash must match; row count must equal the sum of included plus excluded rows
exclusions_summaryCounts by reason code, such as out_of_scope, deletion_request, failed_redaction_checkShows what was held back and why without exposing itCounts must equal ledger rows with each exclusion code
sealed_at2026-10-18T14:00:00ZWhen the release stopped changingMust be later than every extraction window end

What goes in the record ledger so one row can be traced to its permission?

The ledger is one row per candidate record, included or not, and it is what turns a release-level hash into a record-level answer. Without it you can prove a file is unchanged, but not why a particular customer conversation is in that file or what was done to it on the way.

Keep sensitive identifiers out of the ledger itself. Where the native key would reveal a person or an account, store a keyed hash of it and keep the key with the source system owner. That still lets you find every release containing a given source record when a deletion request or a rights question arrives, without making the ledger a second copy of the data you restricted.

The parent digest and the transform chain follow the shape of the W3C PROV data model, which treats a derivation as the transformation of one entity into another and a revision as a derivation that produces a revised version of an original. Recording both lets you say not only that 2.1.1 replaced 2.1.0 but which step made the difference.

What goes in the record ledger so one row can be traced to its permission?
Ledger columnMeaningWhy reconciliation needs it
record_idStable identifier assigned by you, unchanged across releasesLets a corrected record stay the same record instead of looking like a removal plus an addition
source_refKeyed hash of system name plus native key, such as a ticket numberFinds every release containing one source record when a request names it
extract_idThe export job in the source register that read itTies the row to the system state on a specific date
transform_chainDigest of the ordered transformation list applied to this rowShows whether this row went through the current or an older redaction pass
permission_refThe agreement or scope schedule that covers this rowAnswers whether the row was allowed out, and under what
content_sha256Hash of the canonical delivered recordSettles whether a disputed row is the one you shipped
statusincluded, or an exclusion reason codeProves a held-back record was considered and why it stayed home

Worked example: one release that validates and one that fails (illustrative)

The example below is invented to show structure. It does not describe a real company, customer or dataset, and every count is illustrative.

A data owner prepares helpdesk-tickets 2.1.0 from one export job covering a two-year window. The ledger has 48,000 rows: 46,850 included, 900 excluded as out of scope and 250 excluded on failed redaction checks. Files are listed in path order with sizes and hashes. The manifest names the redaction tool and its version, the scope schedule ID from the signed agreement and the 2.0.3 digest as parent. Canonicalized and hashed, the manifest yields a digest whose first twelve characters go into the label as build metadata.

  • Valid release: the validator recomputes every file hash and the ledger hash, confirms 46,850 plus 900 plus 250 equals 48,000, finds that the exclusion counts match the ledger, resolves the scope schedule and the parent digest, and confirms [email protected] maps to one digest. It passes.
  • Failing release, label reuse: two weeks later an engineer fixes 40 tickets with a missed signature block, regenerates the files and publishes them as 2.1.0 again. The validator reports that [email protected] now maps to two digests and refuses the second. The fix ships as 2.1.1, with 2.1.0 as parent and a new transformation step version, and the release note lists the 40 record IDs.
  • Failing release, broken ledger: the regenerated ledger lost permission_ref on 120 rows that came from a newly added queue. The validator reports 120 included rows whose permission reference does not resolve, and the release stays unsealed until the scope question is answered.
  • Failing digest, canonicalization skipped: a second script wrote the same manifest with keys in a different order and got a different digest. Nothing in the content changed; the fix is to hash the JCS form, after which both scripts agree.

Which reconciliation questions the identity answers, and where it stops

The identity answers questions about what was released; it cannot answer questions about what happened afterwards. Know which is which before you promise anything to a buyer or a regulator.

Which reconciliation questions the identity answers, and where it stops
Question someone asksIdentifier that answers itEvidence you produceLimit
Is this the 2.1.0 we were sent?Release digestRecomputed digest over their copy of the manifestProves identity of the manifest and files, not how they were used
Was this specific record in our delivery, unchanged?Record content hash plus record_idLedger row and matching hashNeeds the recipient to produce the record they hold
Which releases contain a person who asked for deletion?source_ref across ledgersList of release digests and the recipients of eachShows exposure; deletion by recipients needs their own evidence
Did the improved redaction pass reach every release?transform_chainReleases and rows still on the older chainDoes not prove the newer pass caught everything
What allowed this row out?permission_refThe resolved agreement and scope scheduleA reference is not a legal opinion; counsel reads the document
Did anything change between two releases?Parent digest and ledger diffRows added, removed and re-hashed with reasonsOnly as complete as the ledger discipline behind it

Where release identity fits when a company licenses records through SourceX

In a licensing arrangement the release identity is how both sides point at the same thing. Under SourceX's process the company signs an agreement with approved scope and terms before any data is delivered, so the scope schedule that permission_ref resolves to already exists when the first manifest is sealed. De-identification is done by SourceX through approved processing routes, which makes the manifest's transformation chain the natural place to record which pass a release went through.

Very large deliveries can stay in the seller's own storage or travel on encrypted drives rather than being hosted by SourceX, and that is exactly when a portable manifest and digest matter most: the identity travels with the bytes wherever they sit. If you are weighing whether your records could be licensed at all, a short company assessment comes first and needs no data upload.

Frequently asked questions

Should the release ID be the hash itself?

Use the hash as the identity and a readable label as the handle people use. A 64-character digest is reliable but unusable in conversation and release notes. Put a short digest prefix in the label's build metadata so people notice mismatches, and compare full digests in every tool.

Is a file checksum enough to version a dataset?

No. File checksums prove bytes arrived intact, but they say nothing about which records were excluded, which redaction pass ran or which permission covered the release. Hash a manifest that includes the file hashes, the record ledger hash, transformation versions and permission references, and use that digest as the release identity.

Why canonicalize JSON before hashing a manifest?

Because the same content can be written with different key order, spacing or number formatting, and each variant hashes differently. RFC 8785 defines a canonical form designed for hashing. Canonicalize, sort order-insensitive arrays such as file lists, then hash, and two independent tools will agree.

What happens to the identity when records are deleted later?

The released identity stays fixed; it describes what went out on that date. Deletions produce a new release whose ledger marks the rows with a deletion reason, and the parent digest links it to the earlier one. Keep the old manifest and ledger metadata so you can show which recipients received the deleted rows.

Can I use a table snapshot ID as the dataset version identifier?

Use it as a source pointer, not as the identity. Snapshot IDs live inside one table's metadata, references can be moved or expire under retention rules, and recipients of exported files cannot recompute them. Record the snapshot ID in the source register and let the manifest digest identify the release.

Sources

  • The OCI image specification defines a digest as a content identifier written as algorithm, colon and encoded hash, recommends SHA-256 and verifies content by recalculating the digest independently. Source
  • RFC 8785, the JSON Canonicalization Scheme, produces a hashable JSON representation by sorting object properties by UTF-16 code units at every level while keeping array order unchanged. Source
  • Semantic Versioning 2.0.0 forbids modifying a released version's contents and ignores build metadata when determining version precedence. Source
  • The Croissant specification (1.0 and current 1.1) recommends MAJOR.MINOR.PATCH for datasets with data-specific meanings for each level and strongly recommends sha256 checksums for all file objects in versioned datasets. Source
  • The Apache Iceberg table specification gives each snapshot a unique ID, treats tags as labels for single snapshots and branches as mutable references, and lets retention rules remove references and expire snapshots. Source
  • The Frictionless Data Resource specification's hash property defaults to MD5 and indicates other algorithms by prefixing the value with the algorithm name. Source
  • The Frictionless Data Package specification says the package version string should conform to Semantic Versioning. Source
  • The W3C PROV data model defines derivation as the transformation of one entity into another and a revision as a derivation that yields a revised version of an original. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies