Systems and records
Reproducible dataset archive packaging: how to get the same manifest twice
By SourceX Editorial · Updated
9 sections
Short answer
To reproduce a dataset archive with the same manifest twice, remove every input that is not the data: sort member names by code point, normalize file names, pin timestamps to one epoch, zero owners and permissions, strip compression headers, and record tool versions. Hash each payload file into a sorted, relative-path manifest. A buyer then repackages the extracted files with the recorded settings and compares hashes.
System export
Dataset packaging
Key takeaways
- Two archives of identical files differ because tar, zip and gzip record order, times, owners and names unless you fix them.
- The payload manifest, not the archive hash, is the contract: it should list every file by relative POSIX path with a SHA-256 hash, sorted by code point.
- A packaging recipe that names the tool, version, options and epoch lets anyone rebuild the archive and compare it without the supplier's systems.
- When archive hashes differ but the payload manifest matches, the data is the same and only the container drifted.
- Some files, such as columnar files that embed their writer's version, should be verified at the payload level, not forced to be byte-identical across tools.
How do you reproduce a dataset archive with the same manifest twice?
You reproduce a dataset archive by making every byte of the package a function of the data and a written recipe, and nothing else. Same files in, same recipe, same tool versions: same archive hash and same manifest out. If the second build differs, the recipe left something to the machine.
Most packaging scripts do not fail on content. They fail on metadata the archiver adds quietly: the order the file system returned names, the time each file was touched, the user who ran the job and the moment the compressor started. Each of those changes between runs and between machines, so the archive hash changes while the data does not.
The fix has two halves. First, pin the container: order, times, ownership, permissions, name encoding and compression header. Second, separate the manifest from the container, so the manifest describes the payload files themselves and stays stable even when a buyer repackages with a different tool.
Which bytes change when the files did not?
Archive drift comes from a short, predictable list of places, and each has a setting that removes it. The table below is the list to work through before calling a package reproducible. GNU tar, for example, does not sort directory entries unless asked, and its manual recommends name sorting to make archives more reproducible. Python's gzip module writes the current time into the stream header unless you pass a fixed mtime.
Two entries are easy to miss. File names that look the same can be stored as different byte sequences: an accented letter may be one precomposed code point (NFC) or a base letter plus a combining mark (NFD), depending on the system that exported it. And sort order can follow locale rules on one machine and raw code points on another.
| Drift source | Where it enters | Fixed setting | How to check |
|---|---|---|---|
| Member order | Directory listing order returned by the file system | Explicit sorted file list by code point; GNU tar --sort=name; Python tarfile sorts recursion since 3.7 | List archive members twice and diff |
| Modification time | Per-member mtime in tar or zip headers | One epoch for all members, or clamp to SOURCE_DATE_EPOCH | Every member shows the same time |
| Access and change times | Extended pax records written by some tar settings | Delete atime and ctime pax records | No atime or ctime keys in pax headers |
| Owner and group | uid, gid, uname and gname of the account running the job | Zero ids, fixed or empty names, numeric owner | Same owner fields on every member |
| Permissions | Umask of the account or container | Set files to one mode and directories to another | Mode column identical on rebuild |
| Compression header | gzip stores a timestamp and can store the input file name | gzip -n, or mtime 0 in Python gzip | Header bytes identical; hash matches |
| ZIP timestamps | date_time per member, year 1980 or later | Fixed date_time on every member at or after 1980-01-01 | Central directory times identical |
| File name encoding | NFC versus NFD names from different operating systems | Normalize every path to NFC before sorting and writing | No path changes when normalized again |
| Stray files | Hidden files and editor or OS artifacts in the staging folder | Package from an explicit file list, never a whole folder | Member count equals manifest line count |
Should the manifest describe the archive or the files inside it?
Describe the files inside it, and record the archive hash separately as a convenience. The payload manifest is what both parties can regenerate from the extracted data with any tool. The archive hash is only reproducible if the buyer uses the same archiver and compressor, and the Reproducible Builds project notes that gzip output can depend on which gzip implementation produced it.
BagIt (RFC 8493) is a good model for the payload manifest. Each manifest line pairs a checksum with a relative file path, the manifest is named for its algorithm, every payload file must appear in it, and tools must support SHA-256 and SHA-512. You do not need to adopt the full BagIt layout to borrow those rules.
Keep a second, small record for the container: archive file name, size, SHA-256, tool name and version, and the options used. If a buyer's rebuild produces a different archive hash but the payload manifest matches line for line, the data is identical and the difference is in packaging, which is a much smaller problem to resolve.
The deterministic packaging procedure, step by step
Run these steps in order for every release. Each step leaves a piece of evidence in the package, so a reviewer can see that it happened without asking. The commands named are examples; any tool that can apply the same fixed settings works.
| Step | Action | Fixed setting | Evidence recorded |
|---|---|---|---|
| 1. Freeze the payload | Copy approved files into a clean staging folder from an explicit list | No globbing of whole folders | File list with count |
| 2. Normalize names | Convert every relative path to Unicode NFC and forward slashes | NFC, POSIX separators, no leading ./ | Normalized path list |
| 3. Normalize text files | Apply one line ending and encoding to CSV, JSON Lines and text | UTF-8, LF line endings, trailing newline rule stated | Encoding and line-ending rule in the recipe |
| 4. Hash the payload | Compute SHA-256 for every file | Stream the bytes exactly as stored | Payload manifest lines |
| 5. Sort the manifest | Order lines by path using Unicode code point order | Code point sort, not locale collation | Sorted manifest file with its own hash |
| 6. Choose the epoch | Set one timestamp for the release, such as the approval date | SOURCE_DATE_EPOCH as integer seconds since 1970 UTC | Epoch value in the recipe |
| 7. Build the archive | Add members from the sorted list with fixed metadata | GNU tar: --sort=name --format=posix --mtime with the epoch --owner=0 --group=0 --numeric-owner, pax delete of atime and ctime | Exact command line and tar version |
| 8. Compress | Compress the tar stream without name or time in the header | gzip -n at a stated level, or another compressor with a stated version | Compressor name, version and level |
| 9. Rebuild | Repeat steps 4 to 8 on a second machine or container from the same payload | Same recipe, same tool versions | Second archive hash and manifest hash |
| 10. Compare and publish | Diff the two manifests and the two archive hashes; ship only if both match | Byte equality, no tolerance | Rebuild log stored with the release |
The manifest both sides can regenerate
A manifest is only useful as a contract if the buyer can produce the same file from the same data. That means writing down the rules that most teams leave implicit. Path rules can follow the Frictionless Data Package standard, which requires POSIX-style paths that are relative to the descriptor and forbids absolute paths, parent references and hidden folders.
Sort order deserves its own line in the specification. A script that sorts raw code points and a shell pipeline that sorts by locale collation can disagree on order for the same names. Pick code point order on NFC-normalized paths, state it, and name the algorithm in the manifest so nobody has to guess.
| Manifest element | Rule | Why it matters | Illustrative value |
|---|---|---|---|
| Algorithm | SHA-256 for every file, named in the file name | Lets both sides pick the same function | manifest-sha256.txt |
| Line format | Lowercase hex hash, two spaces, relative path | Matches common checksum tools so buyers can verify without code | hash then tickets/2025/part-0001.jsonl |
| Paths | Relative POSIX paths, NFC, no ../, no hidden files | Same path bytes on every platform | events/clinic-a/visits.csv |
| Order | Code point order of the normalized path | Removes locale differences | Uppercase sorts before lowercase |
| Encoding | UTF-8 without byte order mark, LF line endings, final newline | The manifest's own hash is stable | Stated in the recipe file |
| Coverage | Every payload file listed once, nothing else | Missing or extra files are caught by count | Lines equal payload file count |
| Container record | Archive name, size, SHA-256, tool, version, options, epoch | Separates container drift from data drift | release-2026-10.tar.gz with GNU tar 1.35 |
| Recipe version | Version or commit of the packaging script | Ties the archive to the code that built it | packaging recipe v3 |
Five checks a buyer runs on the package with no supplier access
A buyer can validate and rebuild the package alone, provided it carries its payload manifest, its container record and its packaging recipe. Nothing in the checks below needs access to the supplier's source systems, credentials or staging servers; they need the delivered files and the recipe text.
Run the checks in order. The first two prove the data is complete and unchanged. The third proves the container can be rebuilt. A failure at the third step alone is a packaging question, not a data question, and it should not hold up acceptance testing of the content.
| Buyer check | How to run it | Passes when | A failure means |
|---|---|---|---|
| Archive integrity | Hash the delivered archive and compare to the container record | Hashes match | The archive changed in transit or the record is wrong |
| Payload completeness | Extract, list files, compare count and paths to the manifest | Same count and same paths | Missing, extra or renamed files |
| Payload integrity | Hash every extracted file and diff against the manifest | No differences | Changed content, or a text file rewritten on extraction |
| Manifest regeneration | Rebuild the manifest from extracted files using the stated rules | Identical bytes to the delivered manifest | Path normalization, sort order or line format differs |
| Container rebuild | Repackage with the recorded tool, version, options and epoch | Archive hash matches the container record | Tool version or an unrecorded option differs |
Illustrative worked example: two builds, one mismatch
This example is illustrative; the dataset and counts are invented to show the method. A supplier packages a support-ticket dataset of 3 resources and 1,240 files. Build A runs on a laptop and build B in a container. The payload manifests match line for line: 1,240 lines each, identical hashes. The archive hashes differ.
The container records show why. Build A used GNU tar with --sort=name and gzip -n. Build B used the same tar options but piped through a gzip run without -n, so its header carried the build time. Rerunning B with -n produced a matching archive hash. The data was never in question, because the payload manifest already matched.
Now the exception. A later release adds files exported on a different workstation, and the buyer's regenerated manifest no longer matches: 14 lines sort differently and their path bytes differ. Those 14 file names contain accented characters stored in NFD form. Normalizing every path to NFC in step 2, before hashing and sorting, restores a byte-identical manifest. The lesson is that name normalization belongs in the recipe, not in a fix applied after a mismatch.
When byte-identical is the wrong target
Byte equality of the archive is worth enforcing when the same team, tools and recipe build every release. It is the wrong target in three situations, and the manifest should then carry the weight on its own.
First, files that embed their writer. The Parquet format has a created_by field for the application and version that wrote the file, so the same table written by two library versions can differ in bytes. Freeze the Parquet files once, hash them as payload and never regenerate them during packaging. Second, encrypted archives or drives: encryption commonly uses fresh random values, so compare hashes of the decrypted payload instead. Third, deliveries that are not archived at all, such as files left in the supplier's object storage for the buyer to copy; there the payload manifest is the whole contract.
Stop and ask for a narrower scope when files change after approval, for example a live table re-exported for each build. No packaging recipe can make a moving source reproducible; freeze the export first, then package it.
What to ask for when you request a packaged dataset
When you scope a licensed dataset, ask for three files alongside the data: the payload manifest with its stated rules, the container record and the packaging recipe. Ask that the supplier run its own rebuild before delivery and include the rebuild log. Those requests cost little at the start and settle most acceptance disputes before they begin.
SourceX sources datasets from U.S. businesses for each buyer request rather than selling from a stock catalog, so packaging rules like these can be written into the request at the outset. Very large deliveries remain in the seller's storage or travel on encrypted drives, which is one more reason to make the payload manifest, not the archive, the thing both sides verify.
Frequently asked questions
Why does my tar.gz have a different hash every time I build it?
Usually because tar recorded different member times or order and gzip wrote the current time into its header. Sort members by name, set one modification time, zero the owner fields and compress with gzip -n. If the hash still differs, compare tool versions between the two builds.
Is gzip -n enough to make compressed output identical across machines?
It removes the stored name and timestamp, which are the common causes. Output can still depend on the gzip implementation and compression level, so record both in the container record. If two implementations disagree, rely on the payload manifest and recompress with the recorded tool.
Should a dataset manifest use MD5 or SHA-256?
Use SHA-256 unless the buyer asks for SHA-512. BagIt requires tools to support both, and either is a sound default for integrity checks on delivered files. Name the algorithm in the manifest file name or in each hash value so nobody has to guess.
Do I need SOURCE_DATE_EPOCH to package a dataset reproducibly?
You need one fixed timestamp for the release, and SOURCE_DATE_EPOCH is the established way to pass it to tools: an integer count of seconds since 1970 UTC. Choosing the approval date or the export cut-off as that value keeps it meaningful to reviewers.
Is a zip file harder to make reproducible than a tar file?
Not harder, but different fields need fixing. Each zip member stores a date and time that cannot be earlier than 1980, so set every member to the same date at or after 1 January 1980, add members in sorted order and fix permissions. Then verify with the same payload manifest.
Sources
- GNU tar performs no directory sorting by default; the --sort=name option sorts entries by name, which makes archive creation more reproducible. Source
- The GNU tar manual's reproducibility section recommends --format=posix, deleting atime and ctime pax records, clamping modification times to a chosen epoch and compressing with gzip --no-name. Source
- Reproducible Builds recommends sorted names, a fixed mtime, owner and group 0 with numeric ids and gzip -n for repeatable archives, and notes gzip output can depend on the implementation. Source
- SOURCE_DATE_EPOCH is an integer number of seconds since 1970-01-01 00:00:00 UTC that build tools use in place of the current time. Source
- Python's gzip module writes the current time into the header unless mtime is given, and mtime 0 produces a stream that does not depend on creation time. Source
- Python's tarfile adds directory entries in sorted order since 3.7, accepts a filter that can reset owner fields, and uses the PAX format by default since 3.8. Source
- Each ZIP member stores a last-modified date and time whose year must be 1980 or later. Source
- BagIt manifests pair checksums with relative file paths, must list every payload file, and tools must support SHA-256 and SHA-512. Source
- Frictionless Data Package paths must be relative POSIX paths and may not be absolute, reference a parent folder or point into hidden folders. Source
- Parquet file metadata includes an optional created_by string naming the application and version that wrote the file. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.