Systems and records
Dataset artifact checksum schema: verify every file your records reference
By SourceX Editorial · Updated
9 sections
Short answer
To verify every artifact a dataset references, list each file once with a stable ID, relative path, media type, byte size and algorithm-prefixed SHA-256 digest, and repeat that digest inside every record that points to it. A validator then checks that references resolve, sizes and digests match, redacted copies link to their originals, and no orphan or withheld file is referenced.
Checklist
Dataset artifact checksum
Key takeaways
- A delivery hash list proves files are intact; only a digest inside each record reference proves the record still points at the bytes it was labeled against.
- Each artifact entry needs a stable ID, relative path, media type, size, algorithm-prefixed digest, role and, for redacted copies, a derived_from link.
- Run checks in order: header, grammar, uniqueness, path safety, resolution, pinning, size, digest, coverage, lineage.
- Ship a failing fixture that breaks one rule per defect so the buyer can prove the validator rejects what it should.
- A portable package validates on an isolated machine; remote artifacts carry URL, size and digest in a fetch list.
How do you verify the integrity of every artifact a dataset references?
Give every referenced artifact one manifest entry with a stable ID, a relative path, a media type, a byte size and an algorithm-prefixed SHA-256 digest, and make each record carry both the artifact ID and that digest. A validator then proves three things: every reference resolves, every listed artifact is present with the declared size and digest, and no unreferenced artifact slips in unexplained.
That is the whole method in one paragraph. The rest of this page turns it into a schema you can hand to a supplier, a worked package built from support tickets with attachments, the checks a buyer's validator should run, and the failure messages that tell you which side of the link broke.
The scope here is narrower than general delivery verification. A delivery manifest proves the files you received are the files that were sent. An artifact manifest proves that the screenshots, PDFs, audio clips and attachments your records point to are the exact bytes the records were labeled against, which is the property a training or evaluation run actually depends on.
Why a file list with hashes still lets broken references through
A file-level manifest answers whether each delivered file is intact; it does not answer whether each row still points at the right file. Records that reference artifacts break in ways a hash list never sees, because the hash list has no idea which row expects which bytes.
Three failure types cover most of what goes wrong. A dangling reference is a record that names an artifact the package does not contain, often because an attachment export failed silently. An orphan is an artifact no record references, which usually means a join key changed or a record was filtered out after the files were copied. A swapped artifact is the subtle one: the reference resolves and the file hashes correctly against the manifest, but the file is a different version than the one the record was labeled against, for example an unredacted screenshot replaced by its redacted copy after annotation.
Archive packaging standards already separate these concerns. BagIt, published as RFC 8493, calls a bag complete when every payload file is listed in a manifest and valid only when every checksum in every manifest verifies, and it requires each payload manifest to list every payload file exactly once. That gives you completeness and validity at file level. The artifact manifest adds the third property, referential integrity between records and files, by putting the digest inside the reference as well as inside the file list.
What should the checksum and artifact validation manifest include?
The manifest needs one entry per artifact with an identity, a location, a size, a digest and a role, plus a short package header that pins the versions. The field set below borrows from three specifications that already solve parts of this problem: the OCI content descriptor, the Git LFS pointer file and the Croissant FileObject.
Keep the manifest as JSON Lines, one artifact per line, so it streams for packages with millions of files and diffs cleanly between releases. The package header lives in a separate small JSON file that also carries the digest of the artifact manifest itself, because a file cannot contain its own hash.
| Field | Level | Rule | Why it exists |
|---|---|---|---|
| schema_version | Package header | Version of this manifest schema, for example 1.0.0 | Lets the validator refuse a manifest written to rules it does not know |
| content_version | Package header | Immutable label for this release of the records and artifacts | A re-delivery and a new release must never share a label |
| artifacts_manifest_digest | Package header | sha256 digest of the artifacts JSON Lines file | Proves the artifact list itself was not edited after sign-off |
| artifact_id | Artifact | Stable, unique, never reused across releases | Records point to it; a reused ID silently rebinds old labels to new bytes |
| path | Artifact | Relative POSIX path from the package root, no leading slash, no .. | Package stays portable and cannot write outside its folder |
| media_type | Artifact | Registered media type such as image/png or application/pdf | Tells the loader how to parse the bytes and lets the validator sniff for mismatches |
| size | Artifact | Exact byte length, non-negative integer | Cheap pre-check that catches truncation before hashing |
| digest | Artifact | algorithm:hex, lowercase, for example sha256 plus 64 hex characters | The content identity every record reference must repeat |
| role and derived_from | Artifact | Role such as attachment, screenshot or audio; derived_from names the source artifact_id for redacted or transcoded copies | Separates originals from derivatives so labels attach to the right version |
| referenced_by_count | Artifact | Number of record references the supplier expects to resolve here | Lets the validator flag orphans and unexpected fan-out without a second export |
How should a record point at an artifact so the link cannot drift?
Put the digest in the reference, not only in the manifest. A record that stores just a path or a URL will happily resolve to whatever file is there today; a record that stores an ID plus the expected digest fails loudly when the bytes change.
This is the pattern content-addressed systems settled on. The OCI image specification makes digest and size required properties of every descriptor, says retrieved content should be verified against the digest when it comes from an untrusted source, and says content whose length does not match the size should not be trusted. Its digest grammar is the algorithm, a colon, then the encoded hash, and for sha256 the encoded part must be 64 lowercase hex characters, with uppercase explicitly disallowed.
Git LFS reaches the same design from the other direction: the repository stores a small pointer file whose oid line is the hash method and the lowercase hex hash, plus a size in bytes, while the large file lives elsewhere. Browsers apply the idea to scripts. W3C Subresource Integrity lets a page declare the expected hash of a fetched resource, requires user agents to support SHA-256, SHA-384 and SHA-512, and allows several hashes on one resource so an algorithm can be retired later without breaking old pages.
For a dataset, the reference object inside each record therefore needs two fields, artifact_id and digest, and optionally the role the record expects. Avoid storing a bare URL as the reference. If the artifact must be fetched remotely, keep the URL in the manifest entry alongside its digest, so moving the file changes one manifest line instead of every record.
A worked package: support tickets with attachments (illustrative)
The package below is invented to show structure. It describes no real company or dataset: an illustrative help-desk export with 3 tickets and 4 artifacts, one of which is a redacted copy of a screenshot. Digests are shortened here for readability; a real manifest carries the full 64 hex characters.
At a glance
A worked package: support tickets with attachments (illustrative)
Read the lines as the files would appear in the package root.
- package.json: schema_version 1.0.0, content_version 2026-10-r1, artifacts_manifest_digest sha256:7c1e...(64 hex), record_files tickets.jsonl, artifacts_file artifacts.jsonl
- artifacts.jsonl line 1: artifact_id att-0001, path artifacts/att-0001.pdf, media_type application/pdf, size 48213, digest sha256:a3f9...(64 hex), role attachment, referenced_by_count 1
- artifacts.jsonl line 2: artifact_id scr-0002, path artifacts/scr-0002.png, media_type image/png, size 302117, digest sha256:51bd...(64 hex), role screenshot_original, withheld true, referenced_by_count 0
- artifacts.jsonl line 3: artifact_id scr-0002r, path artifacts/scr-0002r.png, media_type image/png, size 298540, digest sha256:e08c...(64 hex), role screenshot_redacted, derived_from scr-0002, referenced_by_count 2
- artifacts.jsonl line 4: artifact_id aud-0003, path artifacts/aud-0003.ogg, media_type audio/ogg, size 1180442, digest sha256:9d42...(64 hex), role call_audio, referenced_by_count 1
- tickets.jsonl ticket T-101: attachments list with one reference, artifact_id att-0001 and digest sha256:a3f9...
- tickets.jsonl ticket T-102: screenshots list with one reference, artifact_id scr-0002r and digest sha256:e08c...
- tickets.jsonl ticket T-103: one reference to scr-0002r (the same screenshot reused in a follow-up ticket) and one to aud-0003
Which validator checks run, and in what order?
Run cheap structural checks first, then resolution, then byte checks, so a broken package fails in seconds instead of after hashing terabytes. Every check below needs only the package itself and a standard SHA-256 tool.
The order matters for diagnosis as much as speed. If reference resolution fails, hashing is pointless; if sizes disagree, the digest will too, and the size message is easier for a supplier to act on.
| Check | Rule | Catches | Validator message |
|---|---|---|---|
| V1 header | package.json parses, schema_version is supported, artifacts_manifest_digest matches the artifacts file | Edited or truncated manifest | header: artifacts.jsonl digest mismatch |
| V2 grammar | Every digest is algorithm:hex; sha256 values are 64 lowercase hex characters | Uppercase or unprefixed hashes, MD5 passed off as SHA-256 | artifact scr-0002r: digest not sha256 grammar |
| V3 uniqueness | artifact_id and path unique; no two paths differ only by case | Collisions that overwrite on case-insensitive file systems | path collision: Artifacts/a.png vs artifacts/a.png |
| V4 path safety | Paths relative, no leading slash, no .. segment, inside package root | Path traversal and absolute paths from the supplier's machine | artifact att-0001: path escapes package root |
| V5 resolution | Every record reference names an existing artifact_id | Dangling references from failed attachment exports | T-104 attachments[0]: unknown artifact_id att-0009 |
| V6 pinning | Digest in each reference equals the manifest digest for that artifact_id | Swapped or re-redacted files after labeling | T-102 screenshots[0]: reference digest differs from manifest |
| V7 presence and size | File exists at path and its byte length equals size | Missing files and truncation | aud-0003: size 1048576, expected 1180442 |
| V8 digest | Recomputed digest equals manifest digest | Corruption and silent replacement | att-0001: sha256 mismatch |
| V9 coverage | Reference count per artifact equals referenced_by_count; zero only if withheld or declared standalone | Orphans and unexpected reuse | scr-0002: 1 reference to a withheld original |
| V10 lineage | derived_from points to a listed artifact; withheld originals are never referenced by records | Unredacted originals leaking into training records | T-105 references withheld original scr-0002 |
What does a failing package look like to the validator?
A useful failing example breaks one rule at a time so each message can be tied to one fix. Ship it next to the valid package, labeled as a test fixture, so the buyer can prove the validator rejects what it should before the first real delivery arrives.
The illustrative failing package below changes the worked example in four places. Each change should trip exactly one check, and a validator that passes any of them is not finished.
- Re-redaction after labeling: scr-0002r was re-exported with a wider redaction box, so its bytes and manifest digest changed, but ticket T-102 still carries the old digest. V6 fails on T-102 with a pinning error, which tells the buyer the labels on that ticket were made against different pixels.
- Silent attachment export failure: ticket T-104 references att-0009, which has no manifest entry and no file. V5 fails with an unknown artifact_id; the supplier needs to re-run the export, not re-hash anything.
- Truncated transfer: aud-0003 arrived at exactly 1,048,576 bytes instead of the declared size. V7 fails on size before any hashing starts, which points to the transfer rather than the source system.
- Original leaked into records: ticket T-105 references scr-0002, the withheld unredacted original. V10 fails, and the package should be quarantined until the supplier confirms whether any other record does the same.
- What should pass: the reuse of scr-0002r by both T-102 and T-103 is legitimate, because referenced_by_count declares 2. Fan-out is not an error when the manifest says it is expected.
Can the buyer check and rebuild the package on an isolated machine?
Yes, if the package is self-describing: relative paths, digests in both the manifest and the references, a published schema for the manifest, and no step that needs the supplier's login, database or ticketing system. The buyer should be able to unpack, validate and load the package on an isolated machine.
Remote artifacts are the usual exception, and BagIt's fetch mechanism shows how to handle them cleanly. RFC 8493 lets a bag travel with holes, listing files to be fetched later, and a bag with holes cannot be checked for completeness until those files are in place. The dataset equivalent is a fetch list where each line carries the URL, the size and the expected digest, so a fetched file is held to the same digest as one that shipped inside the package.
Tabular metadata standards already reserve a place for the digest. The Frictionless Data Resource specification has a bytes property and a hash property that defaults to MD5 but accepts other algorithms when the value is prefixed with the algorithm name, such as sha1 followed by a colon. Croissant lets every FileObject carry a sha256 checksum and strongly recommends recording checksums for all used files in versioned datasets. If your buyer already loads Croissant or Data Package metadata, map artifact_id, digest and size into those fields rather than inventing a parallel format, and keep the record-level reference digest as the extra field neither standard requires.
Rebuilding is the stricter test. Ask whether a second engineer, given only the package and the validator, gets the same artifact set, the same reference map and the same pass or fail result. If the answer depends on a call to the supplier, the package is not yet portable.
Putting artifact integrity rules into a licensed data request
The cheapest moment to get artifact integrity is before export, so state the reference format, the digest algorithm and the failing-fixture requirement in the data request itself. Asking for digests after the files have been copied usually means hashing whatever arrived, which proves nothing about what was labeled.
An artifact manifest requirement belongs in the format line of a SourceX data request, next to scope, volume, language, timing and licensing needs; the records are then sourced from U.S. businesses for that request rather than pulled from an inventory. The supplying business signs off on scope and terms before SourceX receives any file, and that sign-off is the point to agree that redacted derivatives, not originals, are what records reference. For very large artifact sets, deliveries stay in the seller's own storage or ship on encrypted drives, so the digest-bearing manifest is what lets the buyer check bytes it did not transfer over a network.
Frequently asked questions
What should the checksum and artifact validation manifest include?
At minimum, a package header with the schema version, the content version and the digest of the artifact manifest, and one entry per artifact with a stable ID, relative path, media type, byte size, algorithm-prefixed digest and role. Add derived_from for redacted or transcoded copies and an expected reference count so orphans and leaks can be detected.
Is a SHA-256 checksum per file enough to verify referenced artifacts?
No. A per-file checksum proves the file matches the manifest, not that the record pointing at it was labeled against those bytes. Repeat the digest inside each record's reference and check that the two agree; that catches swapped and re-redacted artifacts that a file list misses.
Should records reference artifacts by path, URL or ID?
Reference by a stable artifact ID plus the expected digest. Paths and URLs belong in the manifest entry, where moving a file changes one line. A bare URL in a record resolves to whatever is hosted today, so it cannot prove the content is unchanged.
How do you handle redacted copies of screenshots or documents?
Treat each redacted copy as its own artifact with its own digest and a derived_from link to the original. Mark the original as withheld if it is not delivered, and make the validator fail any record that references a withheld original.
Can MD5 hashes from a supplier's export be reused?
Use them as a transport check if that is all the source system produces, but recompute SHA-256 on receipt and make that the reference digest. Label the algorithm in the digest string so a 32-character value is never mistaken for a SHA-256.
Sources
- The OCI image specification makes digest and size required on every content descriptor, defines the algorithm:encoded digest grammar and requires 64 lowercase hex characters for sha256. Source
- Git LFS pointer files record an oid made of the hash method and lowercase hex hash (sha256 only) plus the size in bytes. Source
- Croissant lets each FileObject carry a sha256 checksum and strongly recommends checksums for all used files in versioned datasets. Source
- The Frictionless Data Resource specification defines bytes and hash properties, with MD5 as the default and other algorithms named by a prefix. Source
- W3C Subresource Integrity requires support for SHA-256, SHA-384 and SHA-512 and allows several hashes per resource for algorithm agility. Source
- RFC 8493 (BagIt) defines complete and valid bags, requires each payload manifest to list every payload file exactly once, and allows fetched files that make a bag checkable only once present. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.