Skip to content

Systems and records

Dataset checksum validation limits, measured on 28 dataset snapshots

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Dataset checksum validation proves the bytes you hold are the bytes that were hashed, nothing more. Across 27 tagged releases and the head commit of the public mledoze/countries dataset, checksums flagged 24 of 27 release steps without saying what changed, gave one content three different hashes in five releases, and passed a 5.0.0 CSV in which 189 of 250 rows carry the wrong country code.

Checklist

Dataset checksum validation

This page covers dataset checksum validation limits, measured on 28 dataset snapshots.Illustrative

Key takeaways

  • A SHA-256 match proves a file is the one that was hashed; it cannot show the file was right, means what the manifest says, or comes with permission.
  • In five releases of mledoze/countries, three JSON files with identical content carried three different checksums, so a mismatch alone is not evidence of change.
  • Release 5.0.0's CSV loads in Python's csv module, yet 207 of its 250 rows do not line up with the 72-field header and 189 rows hold the wrong cca2 and cca3 codes.
  • Since release 5.1.0, currency codes changed for six countries, Bulgaria's from BGN to EUR among them, while the changelog's unreleased section lists one item.
  • The first two releases ship no license file, and a manifest that records only hashes verifies them perfectly.
  • Record four layers for every file, bytes, content, meaning in the buyer's reader and rights, and accept a delivery only when all four pass.

What does a dataset checksum prove, and where are its validation limits?

A checksum proves one thing: the bytes you hold are the bytes someone hashed. NIST's Secure Hash Standard, FIPS 180-4, describes message digests as a way to detect whether a message has changed since the digest was generated, and any change produces a different digest with very high probability. That is integrity in transfer and storage, and every delivered file should have it.

The limits follow from the same definition. A checksum computed after an export error certifies the error. It cannot tell a reformatted file from a changed one, a stale generated copy from a fresh one, or a file that comes with permission from one that does not. Dataset checksum validation limits are not a weakness of SHA-256; they mark the edge of what any byte-level test can see.

This page measures that edge on a public dataset at pinned commits and turns the results into a manifest layout and checklist. It sits beside two existing pages, one on reconciling hashes so supplier and buyer prove they hold the same files, and one on carrying a digest for every artifact a record references. Both assume the bytes are the ones you wanted. This page tests whether they are.

Four layers a delivery manifest has to record

Record every delivered file at four layers, because each fails in its own way and needs its own test. A checksum covers only the first:

  • Bytes: path, size and SHA-256 of the file as delivered. This catches truncation, corruption and substitution in transit, and says nothing about content.
  • Content: a fingerprint of the parsed values under a named canonical form, such as JSON with object keys sorted and whitespace removed. This separates a reformatted file from a changed one.
  • Meaning in the buyer's reader: the file parsed with the reader the buyer will actually use, with every row checked against the header and key columns checked against the source of record. This catches files that load but put values under the wrong names.
  • Rights: the license or deal reference that covers the file, its identifier and a hash of the governing text, recorded per release. This catches deliveries whose permission changed, or was never stated, while the data bytes did not.

Twenty-eight states of one public dataset, tested layer by layer

mledoze/countries is a public list of about 250 countries and territories with codes, names, currencies and translations, licensed under the Open Database License and published on GitHub. It keeps a source file, countries.json, and generated copies in a dist folder as JSON, unescaped JSON, CSV, XML and YAML. Its history covers every layer: 27 tagged releases from v1.0 in July 2013 to 5.1.0, whose tag points to a February 2025 commit, plus the head commit of 22 October 2026.

A script read the files at each of those 28 states from a partial clone and ran the four layers' checks without running any repository code. The table is the asset. Its right-hand column is everything a hash-only manifest would have told a recipient; the sections that follow explain each row, and the methods note gives every rule.

Twenty-eight states of one public dataset, tested layer by layer
LayerCheck as runResult across 28 statesWhat the checksum alone reported
BytesSHA-256 of countries.json against the previous stateByte-identical in 3 of 27 steps; changed in the other 24Changed or unchanged, nothing more
ContentCanonical JSON of the source and its two generated JSON copies5 releases where three files share one content under three checksumsThree different files
ContentValue diff by field, release 5.1.0 to headCurrency codes changed for 6 countries, 1 capital moved, Breton names added for 250One changed file
MeaningRelease 5.0.0 CSV rows against its 72-field header207 of 250 rows misaligned; 189 rows hold the wrong cca2 and cca3 codesA valid file
ContentGenerated JSON copy against the source, derived field excludedValues differ in 5 of 24 tagged releases; in 5.1.0 one field is missing for all 250Two valid files
RightsLICENSE file and declared license in each stateNo license file in v1.0 or v1.1; license text replaced in v3.0.0 with the license unchangedSilence, then a change
A two-by-two grid crossing whether a hash check passes or flags a change with whether meaning is intact or differs: 3 byte-identical steps, 5 releases with 3 hashes for 1 content, 24 of 27 steps flagged with no detail, and, highlighted, 189 of 250 CSV rows with wrong codes and 2 releases with no license file.
Measured on 27 tagged releases and the head commit of mledoze/countries; counts from the article's script.

What a changed checksum does and does not tell you

A changed checksum says the bytes differ; it does not say whether anything a model would learn differs. In five releases, v1.5, v1.5.1, v1.6.2, v3.0.0 and v4.0.0, the source countries.json and the two generated JSON copies parse to identical content under three different SHA-256 values. In v4.0.0 the three files are 1,168,387, 647,487 and 519,885 bytes long: the source is indented over 36,819 lines, the generated copies put one entry on each of 250 lines, and only the unescaped copy writes non-ASCII characters directly instead of as 34,239 escape sequences. A hash comparison reports three datasets where there is one.

The canonical form has to be named. The JSON specification, RFC 8259, notes that parsing libraries differ in whether they expose the order of object members, so a content fingerprint should sort keys and drop insignificant whitespace before hashing, and the manifest should state that form so the buyer can recompute it.

The opposite case is the common one. In 24 of the 27 steps between consecutive states, countries.json changed in both bytes and content, and the checksum says only that. A value diff from release 5.1.0 to the head commit shows what a recipient pinned to 5.1.0 holds without knowing it: currency codes changed for six countries, among them Bulgaria from BGN to EUR, Sierra Leone from SLL to SLE, and Curaçao and Sint Maarten from ANG to XCG, while Cuba lost CUC and Zimbabwe's ZWB became ZWG. Equatorial Guinea's capital moved from Malabo to Ciudad de la Paz, and Breton names were added for all 250 entries.

The changelog's unreleased section at the head lists one item: "Serbian 'common' and 'official' name for Hungarian (#570)". A change note is useful, but only a value diff tells the recipient which fields to recheck.

Release 5.0.0: a CSV that loads and misreads 189 country codes

The sharpest failure in the history is a file with nothing wrong at the byte level. Release 5.0.0's dist/countries.csv has a 72-field header built by flattening nested fields, but its rows vary in length: 74 are longer than the header, 133 are shorter and 43 match. Under the cca2 header, 189 of 250 rows hold something other than that country's two-letter code, and the cca3 column fails on 189 rows as well.

Whether anyone notices depends on the reader. Python's csv module read all 250 rows without an error. pandas 3.0.5 with default settings stopped with a tokenizing error, because its documentation defines a bad line as one with too many fields and raises an exception on it by default. Told to skip bad lines, pandas loaded 176 rows, and 155 of them carry the wrong cca2 code, because rows with too few fields are accepted and padded at the end, their values shifted under the wrong headers.

RFC 4180, the informational CSV specification, says each line should contain the same number of fields throughout the file. That one test, run in the buyer's own reader, rejects this file; its SHA-256 verifies it. Release 5.1.0's changelog lists "fix column count in CSV export", and every other state has zero misaligned rows. The same file name also hides a format change: the CSV used semicolons through v1.8.0 and commas from v2.0.0.

Generated copies that drift from their source

A delivery often ships one source file and several generated formats, and each generated file can carry a valid checksum while saying something different from its source. The dist folder's README says it contains generated data. Comparing dist/countries.json with countries.json in the same state, after excluding callingCodes, a field the export code derives from idd, sorts the 25 states that ship both files into three groups.

Five states match exactly. Fourteen differ only because an empty object was written as an empty list somewhere inside an entry, a type difference that changes no value but can trip a typed reader. Six differ in values: v1.6.1, v2.1.0, v4.1.0, v4.1.1, 5.1.0 and the head commit. In 5.1.0 the generated copy lacks the unRegionalGroup field that the source carries for all 250 entries, and at the head it also differs on translations for 28 entries and on currencies for 6.

The test is cheap: parse every derived file, compare its content with the source field by field, name any derived field you exclude, and put the result in the manifest beside the hashes.

The rights layer: a license file is a version too

Permission has its own history, and hashes of data files cannot see it. The first two releases, v1.0 and v1.1, contain no LICENSE file. v1.2 added one stating that the dataset is made available under the Open Database License; package.json declared the same license from v1.5 and composer.json from v1.6.1, and both switched to the identifier ODbL-1.0 in v2.0.0. In v3.0.0 the short notice gave way to the full license text, so the license file got a new checksum while the license stayed the same.

The terms are why the license belongs in the manifest. Open Data Commons' summary of ODbL 1.0 sets attribution, share-alike and keep-open conditions: if you publicly use an adapted version of the database, or a work produced from an adapted version, you must also offer that adapted database under the ODbL. Anyone merging such a table into training data needs that condition recorded against the exact release, and reviewed by counsel for the use in question.

For licensed business data, the equivalent is the deal reference: the agreement and approved scope that cover each file. Record its identifier and a hash of the governing text with every delivery version, so a later reader can tell which permission applied to which bytes.

Versioned delivery manifest and reproducibility checklist

The manifest table adds four kinds of field to the usual path, size and hash, with example values from release 5.1.0; the checklist is the order in which to use them. Fill one manifest row per file per delivery version, and run the checklist before the delivery reaches training storage.

Checklist · 8 items

  1. Hash every delivered file with SHA-256 and compare with the supplier's manifest before opening anything.
  2. Parse each file and compute its content fingerprint under the named canonical form; a hash change with an unchanged fingerprint is format churn, recorded but not treated as new data.
  3. Read every tabular file with the buyer's own reader and version: each row has the header's field count, and code and ID columns match the source of record.
  4. Compare every generated or converted file with its source field by field, naming each derived field you exclude.
  5. Diff values against the last accepted version by field and reconcile the diff with the supplier's change note; an unexplained field change blocks acceptance.
  6. Confirm the rights reference for this version: license or deal identifier, a hash of the governing text, and any attribution or share-alike condition.
  7. Record the reader names and versions, the script and its commit, and the canonical form in the manifest, so a third party can rerun every check and get the same result.
  8. Accept only when all four layers pass, or when each exception is written down and signed off.

0 of 8 ticked. Tick each item as you go.

Versioned delivery manifest and reproducibility checklist
Manifest fieldLayerExample (release 5.1.0)Test that uses it
sha256, bytesBytescountries.json: 1c097ce85e7f (first 12 hex digits), 1,366,674 bytesRecompute on arrival and compare
content_fingerprint, canonical_formContentSHA-256 of the JSON with sorted keys, no whitespace, UTF-8Equal fingerprints mean equal content
derived_from, derived_fieldsContentdist/countries.json from countries.json; callingCodes derived from iddField-by-field comparison with the source
reader_testMeaningCSV, comma-separated, header row, 74 fields in every rowParse in the buyer's reader; check field counts and key columns
value_diff_refMeaningSince 5.0.0: translations changed for 250 entries, unRegionalGroup added for 250, name changed for 11, altSpellings for 8, capital and idd for 1 eachReconcile with the change note
rights_ref, license_sha256RightsODbL-1.0; LICENSE a221a27af5e4 (first 12 hex digits)License present, identifiers agree, conditions recorded

Putting meaning tests into a data request

The cheapest time to get these checks is before anything is exported. Name the reader and version the files must survive, ask for a content fingerprint and a field-level value diff with every version, and require the rights reference to sit beside the hashes. The byte layer is then settled by the reconciliation receipt and per-artifact digests that the two related checksum pages describe.

SourceX sources each dataset on request from the business that holds the records instead of selling from stock, so tests like these can be written into the request together with scope, format and licensing needs. On the supplier side, the supplying company signs off on scope and terms before a single record is delivered, and that signed scope is the rights reference a manifest should point to.

Methods note: commits, rules and what was not measured

Data: the mledoze/countries repository on GitHub, read on 2026-10-28 from a partial clone with git show at its 27 tags in tag-creation order (v1.0 to 5.1.0) and at head commit 6031f5df137c41c76d180c55a0782185ae3ed05b, dated 2026-10-22. The 5.1.0 tag points to a commit dated 2025-02-26, while the changelog dates that release 2024-11-30. No repository code was run. Script: bin/xs0944_checksum_layers.py, with its output in bin/xs0944_output.txt, in the writer's W10-main-029 scratch folder; no data snapshot is kept.

Rules as applied: SHA-256 is computed over raw file bytes. JSON and CSV files are decoded as UTF-8 with any leading byte-order mark removed; countries.json had none in any state. The CSV read is dist/countries.csv where it exists and countries.csv at the repository root otherwise (v1.0 to v1.2). Two JSON files have the same content when json.dumps(parsed, sort_keys=True, ensure_ascii=False, separators=(",", ":")) gives equal strings, so object key order, whitespace and escaping are ignored and array order is kept. Steps compare each state with the previous one. Value diffs key entries by cca3 and compare each top-level field's canonical value. The CSV delimiter is whichever of semicolon or comma occurs more often in the header line; empty rows are dropped; a row is misaligned when its field count differs from the header's; a code cell is wrong when the value under the cca2 or cca3 header differs from that code for the entry at the same position in the same state's countries.json, which has the same number of entries as the CSV in every state.

pandas 3.0.5 read the 5.0.0 CSV once with defaults and once with dtype=str, keep_default_na=False and on_bad_lines='skip', judging each loaded row against the entry whose common name equals its first column. Generated copies are compared by cca3 with callingCodes excluded; a difference counts as type-only when the two values are equal once every empty object and empty list, at any depth, is treated as the same empty value. Rights: presence and SHA-256 of LICENSE, the license or licenses field of package.json, and the license field of composer.json.

Not measured: the XML and YAML copies, commits between tags other than the head, and correctness against ISO 3166 or ISO 4217, since the tests check consistency, not truth. A consumer that depends on key order would see differences this content rule ignores. The currency and capital changes describe the repository's files, not any official register.

Frequently asked questions

Is a matching SHA-256 checksum enough to accept a dataset delivery?

No. A match proves you hold the exact bytes that were hashed, including any export error made before hashing. Add a content fingerprint, a test in the reader you will use, a value diff against the last accepted version and a rights reference before accepting.

Why can two files with the same data have different checksums?

Indentation, key order, Unicode escaping and line endings all change bytes without changing values. In mledoze/countries, five releases ship three JSON files with identical content and three different SHA-256 values. Compare content fingerprints under a named canonical form instead.

What is a content fingerprint for a dataset file?

It is a hash of the parsed values after a stated canonical form, for JSON typically sorted keys, no insignificant whitespace and UTF-8. Two files with equal fingerprints hold the same content even when their bytes differ. The manifest must name the form so the buyer can recompute it.

Should a buyer reject a CSV whose rows have different field counts?

Yes, unless the supplier explains and fixes it. RFC 4180 says each line should contain the same number of fields, and release 5.0.0 of mledoze/countries shows the cost: 207 misaligned rows and 189 wrong codes in a file that Python's csv module reads without complaint.

Where does the license belong in a dataset manifest?

On every delivery version, as an identifier plus a hash of the governing text or deal reference. Permission changes on its own schedule: the first two releases of mledoze/countries have no license file at all, and a later release changed the license text's bytes without changing the license.

Sources

  • The mledoze/countries repository: 27 tagged releases and the head commit, read with git show; source of every count in the results table, the value diff, the CSV test, the generated-copy comparison and the license history. Source
  • FIPS 180-4 specifies hash algorithms whose digests are used to detect whether messages have changed since the digests were generated. Source
  • RFC 8259 notes that JSON parsing libraries differ as to whether they make the ordering of object members visible to calling software. Source
  • RFC 4180 says each line of a CSV file should contain the same number of fields throughout the file. Source
  • pandas read_csv raises an exception on a bad line, defined as a line with too many fields, unless on_bad_lines is changed from its default of 'error'. Source
  • The ODbL 1.0 summary sets attribution, share-alike and keep-open conditions, including offering a publicly used adapted database under the ODbL. Source
  • SourceX sources datasets on request rather than from stock, and buyers describe scope, format and licensing needs in the request. Source
  • The supplying company approves the scope and terms and signs before any data is delivered. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies