Systems and records
Dataset delivery checksum reconciliation: proving both sides match
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Supplier and buyer confirm they hold identical files by each computing SHA-256 hashes of the files they hold and comparing manifests row by row. The supplier signs a manifest of paths, byte sizes and hashes; the buyer recomputes from its stored files, never from console values; every mismatch is resolved by resending and rehashing; and both sign one receipt that later scopes acceptance and deletion.
System export
Dataset delivery checksum
Key takeaways
- Reconciliation compares two independently computed manifests, not one manifest checked once.
- Carry delivery ID, approval reference, algorithm, counts and a signature in the header, and path, bytes and SHA-256 per file.
- Use SHA-256, not MD5, for anything that may serve as evidence.
- Cloud storage checksums for multipart uploads may be composite values that never equal a whole-file hash.
- The signed receipt fixes what was delivered, which later defines acceptance and what must be deleted.
What does it mean for both sides to hold the same dataset?
Both sides hold the same dataset when three things line up: the version the owner approved, the package the supplier sent, and the files the buyer stored. Dataset delivery checksum reconciliation is the step that proves all three match, using a manifest of file hashes that each side computes independently and then compares.
One-sided verification is common: the buyer checks files against the supplier's manifest and moves on. That catches corruption, but it leaves no record the supplier can rely on. If a dispute arises months later about a missing table or an altered file, the supplier needs the buyer's own computed hashes, signed and dated, not just its own claim of what it sent.
So reconciliation adds two things to a normal integrity check. The buyer produces its own manifest from the files it actually stored, and both parties sign one receipt record that lists the result. That record later scopes acceptance, the deletion duty at the end of the license and any audit.
Agree the method before the first file moves. Put the algorithm, the manifest format, who signs, the channel for the manifest and the deadline for the buyer's return manifest into the delivery specification. A buyer asked to produce a signed receipt after the fact may reasonably say no; one that agreed to it in the specification will treat it as part of intake.
Which fields does a checksum reconciliation manifest need?
A reconciliation manifest has a package header and one row per file. The header identifies the delivery and the approval it came from; the file rows carry the evidence. The structure follows the pattern the BagIt packaging standard (RFC 8493) uses for archives: a payload manifest of checksum and relative path for every file, plus a tag manifest that covers the manifests themselves.
The field values below are illustrative and do not describe any real delivery. Adapt names to your own conventions, but keep every field: each one answers a question someone will ask later.
| Header field | Purpose | Illustrative value | Rule |
|---|---|---|---|
| delivery_id | Names this one transfer | DLV-2026-014 | Unique; never reused for a resend |
| dataset_version and approval_ref | Ties the package to the scope the owner approved | v3.2; approval record APR-88 | Must match the signed scope document |
| hash_algorithm | Tells the other side how to recompute | SHA-256 | One algorithm per manifest |
| file_count and total_bytes | Quick completeness check before hashing | 42 files; 18,204,113,920 bytes | A fast first check only, never the verdict |
| generated_at_utc and generated_by | Who produced the manifest and when | 2026-10-18T14:05Z; named engineer | UTC timestamps on both sides |
| manifest_signature | Proves the manifest was not edited in transit | Detached signature file manifest.sig | Signed with a key the other party verified beforehand |
File-level fields and what each one proves
Each file row must let the buyer recompute and compare without asking a question. A hash with no path is useless, and a path with no size forces a full hash before a missing file is even noticed. Record counts are optional but valuable for tabular files, because they explain mismatches that hashes only detect.
Decide early where the hash is taken. If the package travels encrypted, hash the decrypted files, because those are what the buyer stores and what a deletion duty will later cover. If the buyer keeps the files compressed, hash the archive as well and list it in the header, so both forms are traceable to one approval.
| File field | Purpose | Illustrative value | Rule |
|---|---|---|---|
| relative_path | Locates the file inside the package | tickets/2024/part-0007.parquet | Relative to the package root; forward slashes; no paths outside it |
| bytes | Detects truncation before hashing | 436,912,004 | Exact byte count of the stored file |
| sha256 | Detects any change to the file | 64 hex characters, lowercase | Computed on the final file as stored, after encryption is removed |
| record_count | Explains content-level mismatches | 1,204,551 rows | State the counting rule, such as rows excluding a header line |
| status (buyer side only) | The result for this file | match, mismatch, missing or extra | Filled by the buyer from its own computation |
How does the reconciliation procedure run, step by step?
The procedure runs in ten steps, five on each side, and the key rule is that the buyer computes its hashes from its own stored files, never by copying values from the supplier's manifest or a storage console.
Steps one to five belong to the supplier and steps six to ten to the buyer, with the final signature shared. Keep the manifest separate from the data channel where you can, so the same failure cannot alter both.
Checklist · 10 items
- 1. Supplier freezes the approved version and records the approval reference.
- 2. Supplier generates the manifest from the final package, after any compression and before encryption for transport, so the buyer can hash the same bytes after decrypting.
- 3. Supplier signs the manifest with a detached signature and keeps the signature with the delivery record.
- 4. Supplier sends the manifest and signature on a different channel from the data, and shares the public key or its fingerprint in advance.
- 5. Supplier transfers the package and notes the transfer start and end times.
- 6. Buyer verifies the manifest signature before using any value in it.
- 7. Buyer runs the quick check: file count and total bytes against the header.
- 8. Buyer recomputes SHA-256 for every stored file and writes its own manifest with a status per file.
- 9. Buyer sends its manifest back; both sides diff the two manifests and resolve every row that is not a match.
- 10. Both parties sign one receipt record stating the delivery ID, the result and the date, and each keeps a copy.
0 of 10 ticked. Tick each item as you go.
Which hash algorithm and tools should you use?
Use SHA-256 for the manifest and a standard command-line tool to compute and check it. NIST's Secure Hash Standard, FIPS 180-4, defines SHA-256 and states the purpose plainly: digests are used to detect whether messages have changed since the digests were generated, and any change will most likely produce a different digest.
Avoid MD5 for anything that might be used as evidence. RFC 6151 explains that published collision attacks make MD5 unsuitable where collision resistance matters, as with digital signatures. MD5 still catches accidental corruption, but a reconciliation record is meant to settle disputes, so use an algorithm that is not open to that argument. The BagIt standard reflects the same direction: version 1.0 requires tools to support SHA-256 and SHA-512 and keeps MD5 and SHA-1 only for backward compatibility.
For computing and checking, sha256sum from GNU coreutils writes one line per file, and its check option reads a list of checksums and file names and reports whether each matches. Python's hashlib or any tool producing standard hex SHA-256 output works too, as long as both sides use the same algorithm and the same bytes. For signing the manifest, a detached signature such as GnuPG's keeps the manifest readable; the GNU Privacy Handbook notes that both the document and the detached signature are needed to verify it.
Do not reconcile against cloud storage checksums unless you know how they were computed. Amazon S3 offers SHA-256 only as a composite checksum for multipart uploads, derived from per-part values, so it will not equal a plain SHA-256 of the whole file; full-object checksums for multipart uploads are CRC-based. Those values are useful for transfer integrity inside S3, but the manifest should carry hashes of whole files as stored.
Illustrative worked example: one passing and one failing reconciliation
Illustrative example with invented values. A supplier sends delivery DLV-2026-014 with four files. Hash values are shortened here to their first and last four characters for readability; a real manifest carries all 64 characters.
In the passing case, the buyer's manifest matches every row, file count and total bytes agree, the signature verifies, and both sides sign the receipt the same day.
The failing case is the same delivery received a second time after a transfer was interrupted and resumed. The table shows what the buyer's manifest reported. Three of four rows need action, and the record count note explains one of them before anyone opens a file.
| File | Supplier SHA-256 | Buyer SHA-256 | Buyer status | Next step |
|---|---|---|---|---|
| tickets/part-0001.parquet | a91c...7f02 | a91c...7f02 | match | None |
| tickets/part-0002.parquet | 5be0...19d4 | c3a7...0e61 | mismatch; bytes short by 8,388,608 | Resend file; truncated during resumed transfer |
| customers/accounts.csv | 0d44...b2a9 | not found | missing | Supplier confirms file was in the package; resend |
| customers/accounts (1).csv | not in manifest | e817...44c0 | extra | Buyer deletes the stray copy and records it |
| Package totals | 4 files; 1,312,004,883 bytes | 4 files; 1,303,616,275 bytes | count matches, bytes differ | Quick check flagged the failure before hashing |
How do you resolve each kind of discrepancy?
Resolve each discrepancy by its cause, and record the resolution as a new manifest line rather than editing the old one. The table covers the cases that come up in practice. The main habit is to resend the affected files under the same delivery ID with a resend note, then reconcile again, so the final receipt reflects what the buyer actually stores.
Never mark a mismatch as resolved because a later file looks right in a viewer. Only a matching hash closes a row.
Set a time limit for resolution in the delivery specification, such as a number of business days after the buyer reports, and say what happens if it lapses. Without one, a single unresolved row can leave the whole delivery in limbo, with neither acceptance nor rejection on record.
Keep the original failing manifest. A clean final receipt is more credible when the record shows the problem that was found and how it was fixed than when the history has been tidied away.
| Discrepancy | Likely cause | Who acts | Resolution |
|---|---|---|---|
| Hash mismatch, size differs | Truncated or interrupted transfer | Supplier resends; buyer rehashes | Replace the file, reconcile the row again |
| Hash mismatch, size identical | Corruption, or a file regenerated after the manifest | Supplier checks its own copy first | If the supplier copy matches its manifest, resend; if not, issue a new version and manifest |
| Missing file | Skipped in transfer or wrong path | Supplier | Resend to the exact relative path |
| Extra file | Duplicate download or stray working copy | Buyer | Delete it and note the deletion in the receipt |
| Hashes match, record counts differ | Different counting rule, such as a header row or quoted line breaks | Both | Agree the counting rule and correct the note; the file is identical |
| Signature fails | Manifest edited, or wrong key | Both, before any other step | Stop; confirm the key fingerprint by phone and reissue the signed manifest |
| Cloud console checksum differs from manifest | Composite multipart checksum, not a whole-file hash | Buyer | Ignore for reconciliation; rehash the stored file |
What should the signed receipt say, and why keep it?
The signed receipt should state the delivery ID, the dataset version and approval reference, the hash of the final buyer manifest, the result for every file, the date, and the names of the people signing on each side. Attach the two manifests and the resend notes. Keep it short enough that a reviewer can read it in a minute.
Keep it because three later events depend on it. Acceptance terms often start a review period at delivery; the receipt fixes that date and what was delivered. Deletion at the end of a license needs a precise list of what must be deleted; the receipt's file list and hashes are that list. And any dispute about content, such as whether a field was present, starts with the question of which files the buyer received, which the receipt answers.
Store the receipt with the agreement, not with the data. If the data is later deleted, as it should be at the end of a term, the receipt must survive it.
For recurring deliveries, issue one receipt per delivery ID rather than a rolling document. Each receipt then stands on its own, and the set of receipts together is the full record of what the buyer has received over the term, which is exactly the list a deletion or audit request will need.
Which reconciliation results mean the problem is not the transfer?
Stop the delivery, and bring in security, privacy or counsel, when reconciliation shows something other than a transfer problem. Most mismatches are mechanical; the ones below are not, and continuing would turn a technical issue into a contractual or privacy one.
- The manifest signature fails and the key fingerprint cannot be confirmed out of band.
- A file appears that was not in the approved scope, such as a raw export or an unredacted table.
- The buyer reports content-level differences, such as personal data present, while hashes match the approved version, which means the approval itself needs review.
- Repeated mismatches on the same files after resends, which points to a problem in the supplier's pipeline rather than the transfer.
- Either side cannot say where every copy of the package now sits, including transfer staging areas.
Where the receipt fits when SourceX manages a delivery
Delivery and preparation fall within SourceX's role on a licensing deal, and the approval_ref in the manifest should trace back to the scope the record owner put its signature to before any file left.
SourceX does not host multi-terabyte datasets: very large packages remain in the seller's storage or move on encrypted drives, so the signed manifest and receipt become the portable proof of what changed hands. A company weighing whether its records qualify can begin with a brief assessment that asks for no files.
Frequently asked questions
How can a supplier and buyer verify they received identical dataset files?
Each side computes SHA-256 hashes of the files it holds and lists them in a manifest with paths and byte sizes. The buyer computes its own values from the stored files, the two manifests are compared row by row, and both parties sign a receipt recording the result.
Is MD5 good enough for a dataset delivery manifest?
MD5 still catches accidental corruption, but RFC 6151 explains that collision attacks make it unsuitable where collision resistance matters. Because a reconciliation record may be used to settle disputes, SHA-256 is the safer choice.
Why does the S3 checksum not match my SHA-256?
For multipart uploads, Amazon S3 computes SHA-256 as a composite of per-part checksums, so the value differs from a plain hash of the whole file. Reconcile against hashes you compute on the stored file, not against storage console values.
Should the manifest be signed?
Yes, if the receipt may matter later. A detached signature lets the buyer confirm the manifest was not edited in transit, as long as the signing key's fingerprint was confirmed through a separate channel before delivery.
What if the hashes match but row counts differ?
The files are identical, so the difference is in how rows were counted, often a header line or line breaks inside quoted fields. Agree the counting rule, correct the note, and record that the file itself matched.
Sources
- NIST FIPS 180-4 defines SHA-256, a 256-bit digest used to detect whether a message has changed since the digest was generated; any change will most likely produce a different digest. Source
- RFC 8493 (BagIt 1.0) defines payload manifests of checksum and relative file path, tag manifests that cover the manifests, Payload-Oxum as a quick size and count check that does not replace checksum validation, and requires tools to support SHA-256 and SHA-512. Source
- RFC 6151 says published collision attacks make MD5 unsuitable where collision resistance is needed, such as digital signatures, while noting MD5 used only to catch errors is a different case. Source
- The GNU coreutils manual describes the --check option, which reads file names and checksums from a file and reports whether the checksums match the named files. Source
- The GNU Privacy Handbook shows creating a detached signature with --detach-sig and verifying it with --verify, and notes both the document and the detached signature are needed to verify. Source
- Amazon S3 offers SHA-256 only as a composite checksum for multipart uploads, derived from per-part checksums, so it will not equal a plain SHA-256 of the whole file; full-object checksums for multipart uploads are CRC-based. Source
See if your company qualifies
A short company assessment. No data uploads are needed.