Skip to content

Systems and records

Reproducible dataset archive packaging: how to get the same manifest twice

By SourceX Editorial · Updated

9 sections

Short answer

To reproduce a dataset archive with the same manifest twice, remove every input that is not the data: sort member names by code point, normalize file names, pin timestamps to one epoch, zero owners and permissions, strip compression headers, and record tool versions. Hash each payload file into a sorted, relative-path manifest. A buyer then repackages the extracted files with the recorded settings and compares hashes.

System export

Dataset packaging

This page covers reproducible dataset archive packaging: how to get the same manifest twice.Illustrative

Key takeaways

  • Two archives of identical files differ because tar, zip and gzip record order, times, owners and names unless you fix them.
  • The payload manifest, not the archive hash, is the contract: it should list every file by relative POSIX path with a SHA-256 hash, sorted by code point.
  • A packaging recipe that names the tool, version, options and epoch lets anyone rebuild the archive and compare it without the supplier's systems.
  • When archive hashes differ but the payload manifest matches, the data is the same and only the container drifted.
  • Some files, such as columnar files that embed their writer's version, should be verified at the payload level, not forced to be byte-identical across tools.

How do you reproduce a dataset archive with the same manifest twice?

You reproduce a dataset archive by making every byte of the package a function of the data and a written recipe, and nothing else. Same files in, same recipe, same tool versions: same archive hash and same manifest out. If the second build differs, the recipe left something to the machine.

Most packaging scripts do not fail on content. They fail on metadata the archiver adds quietly: the order the file system returned names, the time each file was touched, the user who ran the job and the moment the compressor started. Each of those changes between runs and between machines, so the archive hash changes while the data does not.

The fix has two halves. First, pin the container: order, times, ownership, permissions, name encoding and compression header. Second, separate the manifest from the container, so the manifest describes the payload files themselves and stays stable even when a buyer repackages with a different tool.

Which bytes change when the files did not?

Archive drift comes from a short, predictable list of places, and each has a setting that removes it. The table below is the list to work through before calling a package reproducible. GNU tar, for example, does not sort directory entries unless asked, and its manual recommends name sorting to make archives more reproducible. Python's gzip module writes the current time into the stream header unless you pass a fixed mtime.

Two entries are easy to miss. File names that look the same can be stored as different byte sequences: an accented letter may be one precomposed code point (NFC) or a base letter plus a combining mark (NFD), depending on the system that exported it. And sort order can follow locale rules on one machine and raw code points on another.

Which bytes change when the files did not?
Drift sourceWhere it entersFixed settingHow to check
Member orderDirectory listing order returned by the file systemExplicit sorted file list by code point; GNU tar --sort=name; Python tarfile sorts recursion since 3.7List archive members twice and diff
Modification timePer-member mtime in tar or zip headersOne epoch for all members, or clamp to SOURCE_DATE_EPOCHEvery member shows the same time
Access and change timesExtended pax records written by some tar settingsDelete atime and ctime pax recordsNo atime or ctime keys in pax headers
Owner and groupuid, gid, uname and gname of the account running the jobZero ids, fixed or empty names, numeric ownerSame owner fields on every member
PermissionsUmask of the account or containerSet files to one mode and directories to anotherMode column identical on rebuild
Compression headergzip stores a timestamp and can store the input file namegzip -n, or mtime 0 in Python gzipHeader bytes identical; hash matches
ZIP timestampsdate_time per member, year 1980 or laterFixed date_time on every member at or after 1980-01-01Central directory times identical
File name encodingNFC versus NFD names from different operating systemsNormalize every path to NFC before sorting and writingNo path changes when normalized again
Stray filesHidden files and editor or OS artifacts in the staging folderPackage from an explicit file list, never a whole folderMember count equals manifest line count

Should the manifest describe the archive or the files inside it?

Describe the files inside it, and record the archive hash separately as a convenience. The payload manifest is what both parties can regenerate from the extracted data with any tool. The archive hash is only reproducible if the buyer uses the same archiver and compressor, and the Reproducible Builds project notes that gzip output can depend on which gzip implementation produced it.

BagIt (RFC 8493) is a good model for the payload manifest. Each manifest line pairs a checksum with a relative file path, the manifest is named for its algorithm, every payload file must appear in it, and tools must support SHA-256 and SHA-512. You do not need to adopt the full BagIt layout to borrow those rules.

Keep a second, small record for the container: archive file name, size, SHA-256, tool name and version, and the options used. If a buyer's rebuild produces a different archive hash but the payload manifest matches line for line, the data is identical and the difference is in packaging, which is a much smaller problem to resolve.

The deterministic packaging procedure, step by step

Run these steps in order for every release. Each step leaves a piece of evidence in the package, so a reviewer can see that it happened without asking. The commands named are examples; any tool that can apply the same fixed settings works.

The deterministic packaging procedure, step by step
StepActionFixed settingEvidence recorded
1. Freeze the payloadCopy approved files into a clean staging folder from an explicit listNo globbing of whole foldersFile list with count
2. Normalize namesConvert every relative path to Unicode NFC and forward slashesNFC, POSIX separators, no leading ./Normalized path list
3. Normalize text filesApply one line ending and encoding to CSV, JSON Lines and textUTF-8, LF line endings, trailing newline rule statedEncoding and line-ending rule in the recipe
4. Hash the payloadCompute SHA-256 for every fileStream the bytes exactly as storedPayload manifest lines
5. Sort the manifestOrder lines by path using Unicode code point orderCode point sort, not locale collationSorted manifest file with its own hash
6. Choose the epochSet one timestamp for the release, such as the approval dateSOURCE_DATE_EPOCH as integer seconds since 1970 UTCEpoch value in the recipe
7. Build the archiveAdd members from the sorted list with fixed metadataGNU tar: --sort=name --format=posix --mtime with the epoch --owner=0 --group=0 --numeric-owner, pax delete of atime and ctimeExact command line and tar version
8. CompressCompress the tar stream without name or time in the headergzip -n at a stated level, or another compressor with a stated versionCompressor name, version and level
9. RebuildRepeat steps 4 to 8 on a second machine or container from the same payloadSame recipe, same tool versionsSecond archive hash and manifest hash
10. Compare and publishDiff the two manifests and the two archive hashes; ship only if both matchByte equality, no toleranceRebuild log stored with the release

The manifest both sides can regenerate

A manifest is only useful as a contract if the buyer can produce the same file from the same data. That means writing down the rules that most teams leave implicit. Path rules can follow the Frictionless Data Package standard, which requires POSIX-style paths that are relative to the descriptor and forbids absolute paths, parent references and hidden folders.

Sort order deserves its own line in the specification. A script that sorts raw code points and a shell pipeline that sorts by locale collation can disagree on order for the same names. Pick code point order on NFC-normalized paths, state it, and name the algorithm in the manifest so nobody has to guess.

The manifest both sides can regenerate
Manifest elementRuleWhy it mattersIllustrative value
AlgorithmSHA-256 for every file, named in the file nameLets both sides pick the same functionmanifest-sha256.txt
Line formatLowercase hex hash, two spaces, relative pathMatches common checksum tools so buyers can verify without codehash then tickets/2025/part-0001.jsonl
PathsRelative POSIX paths, NFC, no ../, no hidden filesSame path bytes on every platformevents/clinic-a/visits.csv
OrderCode point order of the normalized pathRemoves locale differencesUppercase sorts before lowercase
EncodingUTF-8 without byte order mark, LF line endings, final newlineThe manifest's own hash is stableStated in the recipe file
CoverageEvery payload file listed once, nothing elseMissing or extra files are caught by countLines equal payload file count
Container recordArchive name, size, SHA-256, tool, version, options, epochSeparates container drift from data driftrelease-2026-10.tar.gz with GNU tar 1.35
Recipe versionVersion or commit of the packaging scriptTies the archive to the code that built itpackaging recipe v3

Five checks a buyer runs on the package with no supplier access

A buyer can validate and rebuild the package alone, provided it carries its payload manifest, its container record and its packaging recipe. Nothing in the checks below needs access to the supplier's source systems, credentials or staging servers; they need the delivered files and the recipe text.

Run the checks in order. The first two prove the data is complete and unchanged. The third proves the container can be rebuilt. A failure at the third step alone is a packaging question, not a data question, and it should not hold up acceptance testing of the content.

Five checks a buyer runs on the package with no supplier access
Buyer checkHow to run itPasses whenA failure means
Archive integrityHash the delivered archive and compare to the container recordHashes matchThe archive changed in transit or the record is wrong
Payload completenessExtract, list files, compare count and paths to the manifestSame count and same pathsMissing, extra or renamed files
Payload integrityHash every extracted file and diff against the manifestNo differencesChanged content, or a text file rewritten on extraction
Manifest regenerationRebuild the manifest from extracted files using the stated rulesIdentical bytes to the delivered manifestPath normalization, sort order or line format differs
Container rebuildRepackage with the recorded tool, version, options and epochArchive hash matches the container recordTool version or an unrecorded option differs

Illustrative worked example: two builds, one mismatch

This example is illustrative; the dataset and counts are invented to show the method. A supplier packages a support-ticket dataset of 3 resources and 1,240 files. Build A runs on a laptop and build B in a container. The payload manifests match line for line: 1,240 lines each, identical hashes. The archive hashes differ.

The container records show why. Build A used GNU tar with --sort=name and gzip -n. Build B used the same tar options but piped through a gzip run without -n, so its header carried the build time. Rerunning B with -n produced a matching archive hash. The data was never in question, because the payload manifest already matched.

Now the exception. A later release adds files exported on a different workstation, and the buyer's regenerated manifest no longer matches: 14 lines sort differently and their path bytes differ. Those 14 file names contain accented characters stored in NFD form. Normalizing every path to NFC in step 2, before hashing and sorting, restores a byte-identical manifest. The lesson is that name normalization belongs in the recipe, not in a fix applied after a mismatch.

When byte-identical is the wrong target

Byte equality of the archive is worth enforcing when the same team, tools and recipe build every release. It is the wrong target in three situations, and the manifest should then carry the weight on its own.

First, files that embed their writer. The Parquet format has a created_by field for the application and version that wrote the file, so the same table written by two library versions can differ in bytes. Freeze the Parquet files once, hash them as payload and never regenerate them during packaging. Second, encrypted archives or drives: encryption commonly uses fresh random values, so compare hashes of the decrypted payload instead. Third, deliveries that are not archived at all, such as files left in the supplier's object storage for the buyer to copy; there the payload manifest is the whole contract.

Stop and ask for a narrower scope when files change after approval, for example a live table re-exported for each build. No packaging recipe can make a moving source reproducible; freeze the export first, then package it.

What to ask for when you request a packaged dataset

When you scope a licensed dataset, ask for three files alongside the data: the payload manifest with its stated rules, the container record and the packaging recipe. Ask that the supplier run its own rebuild before delivery and include the rebuild log. Those requests cost little at the start and settle most acceptance disputes before they begin.

SourceX sources datasets from U.S. businesses for each buyer request rather than selling from a stock catalog, so packaging rules like these can be written into the request at the outset. Very large deliveries remain in the seller's storage or travel on encrypted drives, which is one more reason to make the payload manifest, not the archive, the thing both sides verify.

Frequently asked questions

Why does my tar.gz have a different hash every time I build it?

Usually because tar recorded different member times or order and gzip wrote the current time into its header. Sort members by name, set one modification time, zero the owner fields and compress with gzip -n. If the hash still differs, compare tool versions between the two builds.

Is gzip -n enough to make compressed output identical across machines?

It removes the stored name and timestamp, which are the common causes. Output can still depend on the gzip implementation and compression level, so record both in the container record. If two implementations disagree, rely on the payload manifest and recompress with the recorded tool.

Should a dataset manifest use MD5 or SHA-256?

Use SHA-256 unless the buyer asks for SHA-512. BagIt requires tools to support both, and either is a sound default for integrity checks on delivered files. Name the algorithm in the manifest file name or in each hash value so nobody has to guess.

Do I need SOURCE_DATE_EPOCH to package a dataset reproducibly?

You need one fixed timestamp for the release, and SOURCE_DATE_EPOCH is the established way to pass it to tools: an integer count of seconds since 1970 UTC. Choosing the approval date or the export cut-off as that value keeps it meaningful to reviewers.

Is a zip file harder to make reproducible than a tar file?

Not harder, but different fields need fixing. Each zip member stores a date and time that cannot be earlier than 1980, so set every member to the same date at or after 1 January 1980, add members in sorted order and fix permissions. Then verify with the same payload manifest.

Sources

  • GNU tar performs no directory sorting by default; the --sort=name option sorts entries by name, which makes archive creation more reproducible. Source
  • The GNU tar manual's reproducibility section recommends --format=posix, deleting atime and ctime pax records, clamping modification times to a chosen epoch and compressing with gzip --no-name. Source
  • Reproducible Builds recommends sorted names, a fixed mtime, owner and group 0 with numeric ids and gzip -n for repeatable archives, and notes gzip output can depend on the implementation. Source
  • SOURCE_DATE_EPOCH is an integer number of seconds since 1970-01-01 00:00:00 UTC that build tools use in place of the current time. Source
  • Python's gzip module writes the current time into the header unless mtime is given, and mtime 0 produces a stream that does not depend on creation time. Source
  • Python's tarfile adds directory entries in sorted order since 3.7, accepts a filter that can reset owner fields, and uses the PAX format by default since 3.8. Source
  • Each ZIP member stores a last-modified date and time whose year must be 1980 or later. Source
  • BagIt manifests pair checksums with relative file paths, must list every payload file, and tools must support SHA-256 and SHA-512. Source
  • Frictionless Data Package paths must be relative POSIX paths and may not be absolute, reference a parent folder or point into hidden folders. Source
  • Parquet file metadata includes an optional created_by string naming the application and version that wrote the file. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies