Systems and records
Dataset integrity after decompression: were all the files preserved?
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Verify an extraction against an independent manifest made at source, in both directions. Every listed file must exist with a matching SHA-256 hash, no unlisted file may appear, and total bytes and file count must match. A clean exit code is not proof: duplicate names, case-only differences, long paths, name encoding and refused links can all drop or alter files without an error.
Document stack
Dataset integrity
Key takeaways
- An extraction that finishes without errors can still drop, merge or rename files.
- Duplicate member names, case-only differences and path length limits are easy silent losses to miss.
- Checksum verification only checks listed files, so a reverse pass must look for extra files on disk.
- Inspect the archive listing before extracting and extract into a new, empty folder.
- A shared receipt with the manifest hash and totals lets both parties confirm the same package.
How do you verify that every file survived an archive extraction?
Compare what landed on disk against an independent manifest made before the archive was built, in both directions. The manifest lists every file's relative path, size and SHA-256 hash, plus totals for file count and bytes. After extraction, every manifest entry must exist with a matching hash, nothing unlisted may appear, and the totals must agree.
A successful exit code from unzip, tar or 7-Zip answers a narrower question: the tool did not hit an error it chose to report. Python's zipfile documentation lists the ways extraction goes wrong without the archive being corrupt, including file system limits on characters, name and path length, file size and number of files, interruptions that leave an incomplete extraction, and extracting the same archive twice, which overwrites files without asking.
For a dataset delivery, the difference matters. A licensed archive that silently lost 0.1 percent of its files will still load, still pass a spot check and still train a model; it simply is not the dataset both parties approved.
Why a clean extraction can still lose or alter files
Files go missing during extraction mainly because the archive format and the destination file system disagree about what a valid, unique file name is. The archive may hold two members that the destination treats as one, names the destination cannot store, or entries that a safe extractor refuses on purpose.
The table lists the failure modes worth testing for, each with the signal it leaves and the check that catches it.
| Failure mode | What you see afterward | Check that catches it | Prevention |
|---|---|---|---|
| Duplicate member names | Fewer files than members; the later copy silently replaced the earlier one | Count unique member paths in the listing before extracting | Reject archives with duplicate paths or rename at source |
| Case-only name differences | Report.pdf and report.pdf become one file on a case-insensitive volume | Case-fold the member list and look for duplicates | Rename at source or extract to a case-sensitive volume |
| Paths too long for the destination | Errors or skipped files deep in nested folders on Windows | Measure the longest destination path before extracting | Shorten folder names or enable long path support on the receiving side |
| Name encoding mismatch | Accented or non-Latin names arrive garbled and no longer match the manifest | Compare manifest paths to extracted paths after Unicode normalization | Write archives with UTF-8 names and record the encoding |
| Sanitized or refused entries | Absolute paths and .. segments stripped; links outside the folder refused | Diff the member list against the extracted tree | Use relative paths only and avoid links in deliveries |
| Interrupted or full-disk extraction | A truncated last file and a missing tail of the archive | Total bytes and per-file hashes against the manifest | Check free space first and extract into a new empty folder |
| Missing volume of a split archive | Extraction stops at the boundary of the absent part | Verify each part's hash against a parts list before extracting | Ship a parts list with sizes and hashes |
What should the archive integrity and file count validation guide include?
It should cover four moments: building the manifest at source, testing the archive itself, inspecting the listing before extraction, and reconciling the extracted tree afterward. Each stage produces a small piece of evidence the recipient keeps. The guide below is written to be used as is; replace the tool names with your own where they differ.
| Stage | Check | Pass rule | Evidence to keep |
|---|---|---|---|
| 1 At source | Generate a manifest of relative path, size and SHA-256 for every regular file, plus totals | Manifest covers every file in the approved scope and nothing else | Manifest file and its own SHA-256, sent separately from the archive |
| 2 At source | Record counts by file type and the total bytes and file count as one summary line | Counts match the scope document for the release | Summary line in the release notes |
| 3 Archive test | Run the format's own test, such as a CRC test of every ZIP member or a full tar listing | Every member reads back without error | Test output with date and tool version |
| 4 Before extraction | Inspect the listing: member count, duplicate paths, case-folded duplicates, longest path, absolute or .. paths, links, special files | No duplicates, no unsafe paths, no links unless agreed, member count equals manifest count | Listing report |
| 5 Extraction | Extract into a new, empty folder on a volume with enough free space | Tool exits cleanly and the folder held nothing beforehand | Command, destination and free space before and after |
| 6 Forward reconciliation | Verify every manifest line against the extracted files | Every listed file present with a matching hash | Verification output |
| 7 Reverse reconciliation | List every regular file on disk that is not in the manifest | Nothing extra, or only items on the agreed exclusion list | List of extras, ideally empty |
| 8 Totals | Recompute total bytes and file count on disk | Both equal the summary line from stage 2 | Recomputed summary line |
| 9 Receipt | Record the manifest hash, totals, results and who ran the checks | Both parties hold the same receipt | Signed or emailed receipt filed with the agreement |
Why checksum verification alone misses extra and absent files
Hash verification only checks the files the manifest names, so it needs a second, reverse pass. The GNU coreutils manual describes check mode as reading each line of the checksum file and computing the named file's checksum; a file on disk that is not listed is never examined. Its ignore-missing option goes further and stops reporting listed files that are absent, which is useful for partial downloads and dangerous for acceptance testing.
GNU tar's compare operation has the same blind spot from the other side. The tar manual says it reports archive members that differ from or are missing on disk, but ignores files on disk that have no archive member. Neither tool tells you that the extracted folder contains something nobody approved.
The BagIt packaging format, published as RFC 8493, handles the count problem with a single Payload-Oxum value: total payload bytes and number of payload files written as two numbers. The RFC treats it strictly as a quick test for an incomplete package and still requires full checksum validation before a package is called valid. A delivery can borrow the same idea without adopting the whole format.
Inspecting the listing before anything is written to disk
Many extraction losses are visible in the archive listing, so read it first and decide whether to extract at all. Python's tarfile documentation makes the same point for untrusted archives and lists checks that suit any delivery: extract into a new temporary directory, limit the number and total size of files, check for names that would be shadowed on case-insensitive file systems, and remember that tar files may contain multiple versions of the same file, with later ones overwriting earlier ones.
Since Python 3.14, tar extraction applies a data filter by default, refusing links to absolute paths or outside the destination and special files such as devices. That protects the recipient, but it also means a refused entry is a missing file. Python's zipfile module similarly strips absolute paths and .. components from member names, so a file can land somewhere other than where the manifest expects.
- Member count compared with the manifest count, before extraction.
- Duplicate paths, both exact and after case folding and Unicode normalization.
- Longest destination path, measured with the real target folder prefix included.
- Absolute paths, .. segments, symbolic links, hard links and device entries.
- Total uncompressed size compared with free space on the destination volume.
- Filename encoding flag or declared encoding for ZIP members created on older tools.
Worked example: a document archive moved from Linux to a Windows review laptop
The counts in this example are illustrative. An engineering firm packages 182,406 project files from a Linux file server into a single ZIP for a licensing review. The extraction on a reviewer's Windows laptop finishes without an error dialog, and the folder looks complete.
Forward reconciliation fails on 149 manifest lines. Sixty-one are case-only pairs such as Drawing-A.dwg and drawing-a.dwg, where the second file overwrote the first. Eighty-eight sit in folders nested so deeply that the full path exceeded what the laptop allowed; Microsoft documents a default maximum path of 260 characters for many Windows file functions unless long path support is enabled by both the system and the application.
Reverse reconciliation finds 4 extra files: thumbnail caches the server's desktop client had written into project folders. They were outside the approved scope and were added to the exclusion rules for the next build rather than delivered.
The fix took one rebuild. The firm renamed the case-only pairs at source, flattened the deepest folders with a path map recorded in the manifest, and the reviewer extracted to a short root folder. The second pass matched all 182,406 files, the totals line matched exactly, and both sides filed the same receipt.
How sender and recipient confirm the same package was received
They confirm it by agreeing on the manifest before transfer and exchanging a receipt after reconciliation. The sender sends the manifest's own hash through a different channel from the archive, for example in the signed release email while the archive moves by secure transfer or encrypted drive, so a tampered archive cannot carry a matching tampered manifest.
Access limits belong in the same record. Note who may extract the archive, on which machine, whether the decryption key traveled separately from the data, when download links or drive access expire, and when copies held for transit will be deleted. The recipient's receipt then states the manifest hash, the totals, any exclusions and the date, and the sender files it with the agreement for that release.
When counts are meant to differ, and when to stop and re-send
Some differences are expected and should be written down rather than argued about later. Empty directories are not files and usually do not appear in a file manifest. Operating system debris such as .DS_Store, Thumbs.db or __MACOSX folders can be excluded at source by rule. Hard links stored once in a tar archive may extract as several paths to one file. Each of these belongs on an exclusion or equivalence list that both parties accept before transfer.
Other findings mean stop. A single hash mismatch on a listed file means the bytes differ and the file should be re-sent, not repaired by hand. An unexplained count difference after normalization means the inventory is unknown. Entries that point outside the destination, or archives that expand far beyond their declared size, call for a security review before anyone opens them again.
Applying this to a licensed delivery through SourceX
The same reconciliation applies when records are licensed rather than simply archived, because the agreement describes a specific scope and the delivery has to match it. With SourceX, the company approves the scope and terms and signs before any data is delivered, and SourceX does not host large deliveries, which stay in the seller's storage or move on encrypted drives, so the manifest and receipt steps above fit the way files actually move. Checking eligibility starts with a short company assessment and no data uploads.
Frequently asked questions
Does a ZIP test or unzip -t prove nothing was lost?
It proves each member in the archive reads back and matches its stored CRC-32. It does not prove the archive contains every file from the source, or that the destination stored every member under its own name. You still need a source manifest and a reconciliation after extraction.
Is CRC-32 enough to verify a dataset delivery?
CRC-32 catches accidental corruption inside the archive, which is its job. It is short and not designed to resist deliberate changes, so delivery manifests normally use SHA-256 per file. Use the archive's CRC test as a quick first pass and the manifest hashes for acceptance.
Why does Windows show fewer files than the archive listing?
The usual causes are names that differ only in letter case, which collapse on a case-insensitive volume, and paths longer than the default Windows limit. Duplicate member names in the archive also leave only the last copy. Compare the case-folded listing and the longest path before extracting.
Should I compare file sizes or hashes after extraction?
Use both, for different reasons. Total bytes and file count give a fast signal that something is missing, while per-file hashes prove each file's content is the one that was approved. Sizes alone cannot catch a file that was replaced by another of the same length.
Do empty folders count as missing files?
Not in a file manifest, which lists regular files only. If folder structure matters for your use, for example for an archive whose empty project folders carry meaning, list directories separately or add a placeholder file and record the rule.
Sources
- Python's zipfile documentation lists extraction pitfalls including file system limits, interruption and overwriting on repeat extraction, strips absolute paths and .. components on extract, and stores a CRC-32 per member. Source
- Python's tarfile documentation sets the data extraction filter as the default from 3.14, refusing links outside the destination and special files, and advises extracting into a new directory, limiting file counts and sizes, and checking for case-insensitive shadowing and multiple versions of one file. Source
- The GNU coreutils manual describes checksum verification as checking each file named in the checksum list, and documents an ignore-missing option that stops reporting absent files. Source
- The GNU tar manual says the compare operation reports archive members that differ or are missing on disk and ignores files on disk that have no archive member. Source
- RFC 8493 defines BagIt's Payload-Oxum as total payload bytes and payload file count for a quick completeness test, and still requires full checksum validation before a bag is valid. Source
- Microsoft documents a default maximum path of 260 characters for Windows file functions, with longer paths requiring a system setting and an application opt-in. Source
See if your company qualifies
A short company assessment. No data uploads are needed.