Systems and records
Machine-readable dataset changelog: which tasks changed and why
By SourceX Editorial · Updated
9 sections
Short answer
A dataset changelog should disclose which tasks changed in each release, the type of change (added, removed, relabeled, edited or moved between splits), the reason, and whether results on the old version remain comparable. Make it machine readable at two levels: one entry per changed record keyed by a stable ID with task IDs and hashes, and a release summary with versions, counts and file checksums that a validator can recompute.
System export
Dataset changelog machine
Key takeaways
- Record counts cannot show that evaluation items now have different correct answers; task-level entries can.
- Keep a human release summary, but add one structured entry per changed record for pipelines.
- For task datasets, a corrected label or a split move is a major version, as Croissant classifies it.
- Map release fields to DCAT, Croissant and Data Package; only the affected-task field is new.
- A buyer should be able to replay the change file against the previous version and reproduce the new one.
- Never put removed personal data in the changelog; record IDs, reasons and hashes instead.
What should a dataset changelog disclose about changed tasks?
It should say, for every release, which tasks changed, how each one changed, why, and whether the change breaks comparison with results on the previous version. A task here is the unit a model is trained or scored on: a labeled example, an evaluation item, a trajectory with its expected outcome. Record counts alone cannot tell a buyer that 300 test items now have a different correct answer.
A machine-readable dataset changelog needs two layers. The record level holds one entry per changed record, keyed by a stable ID, with the task IDs it feeds, a change type, a reason code and before and after hashes. The release level summarizes those entries, names the previous version and carries file checksums. The release summary should be computable from the record entries, so a buyer can check one against the other without access to the supplier's systems.
Why isn't a human-readable changelog enough for training pipelines?
Because its own conventions say it is written for people. Keep a Changelog, the most widely copied format, states as its first guiding principle that changelogs are for humans, not machines. It groups changes under Added, Changed, Deprecated, Removed, Fixed and Security, keeps an Unreleased section at the top and marks pulled versions with a bracketed YANKED tag, which it notes is also easier to parse programmatically.
Those headings are a good release-level summary and worth keeping. What they cannot do is drive a pipeline. A data loader needs to know which record IDs to drop from a cached training set, an evaluation harness needs to know which items to rescore, and a contamination check needs to know which records moved from training to test. Each of those needs a structured entry per record, not a sentence per release.
The Datasheets for Datasets questions point the same way. The maintenance section asks whether there is an erratum, whether the dataset will be updated to correct labeling errors or add and delete instances, and how obsolescence of older versions will be communicated. A record-level changelog is the most direct way to answer those questions with evidence.
Which version number should each kind of change get?
Use the Croissant rule for task datasets: any edit to existing data is a major change. Croissant recommends MAJOR.MINOR.PATCH following Semantic Versioning, with PATCH meaning the data is the same but serialized or packaged differently, MINOR meaning existing data is unchanged and still retrievable while new data or fields are added, and MAJOR meaning existing data was edited, removed or shuffled across splits.
Other conventions disagree on corrections. The Frictionless Data Package version recipe treats correcting errors in existing data as a PATCH, and treats adding, removing or reordering fields as MAJOR. That works for reference tables, but for a training or evaluation set a corrected label changes what the task asks the model to produce, so scores before and after are not comparable. Semantic Versioning's own rule that released contents must not be modified, and any change must be a new version, applies under either convention.
| Change | Croissant 1.1 | Data Package recipe | Recommended for task datasets |
|---|---|---|---|
| Re-serialize or repackage the same records | PATCH | No rule; relocation alone needs no version change | PATCH, with new file checksums |
| Add new records in a new split | MINOR | MINOR (add new data) | MINOR; list added task IDs |
| Correct a label or expected answer | MAJOR (existing data edited) | PATCH (correct errors in existing data) | MAJOR; flag affected evaluation items |
| Remove records | MAJOR | MAJOR if referenced data is deleted | MAJOR; reason code required |
| Move records between train and test | MAJOR (shuffled across splits) | Not addressed | MAJOR; flag both splits |
| Add, remove or reorder fields | Depends on whether old data stays retrievable | MAJOR | MAJOR unless purely additive and optional |
| Edit descriptive metadata only | Use the separate metadata version | PATCH | Bump the metadata version, not the content version |
The record-level and release-level change format
This format is the asset. The record-level entries go in one JSON Lines file per release, one object per changed record; the release-level entry goes in the dataset's metadata file. Values in the example column are illustrative and do not describe a real dataset. The validator rule is what a buyer's intake job checks.
Two design choices make the format work. First, the record ID must survive every release; if the supplier regenerates IDs on export, a relabel looks like a removal plus an addition and the task link is lost. Second, the hashes must be computed over a canonical form of each record, with fields in a fixed order, consistent number and date formatting and no volatile fields such as export timestamps. Otherwise every record appears changed after a harmless re-export and the change file becomes noise.
Keep the change_type list short and closed. Six values cover nearly every edit a task dataset sees, and a closed list lets the validator reject anything unexpected. When a record has two kinds of change in one release, such as an input edit and a relabel, write two entries with the same record_id rather than inventing a combined type.
| Field | Level | Allowed values or type | Example (illustrative) | Validator rule |
|---|---|---|---|---|
| change_id and record_id | Record | Strings; record_id is the stable ID used in every release | chg-2.0.0-00041, rec-88213 | record_id exists in the previous or new manifest |
| task_ids | Record | List of task or evaluation item IDs fed by the record | triage-train-5521, triage-eval-0907 | Every ID exists in the task index |
| change_type | Record | added, removed, relabeled, input_edited, split_moved, field_changed | relabeled | Type is consistent with the before and after hashes |
| fields_changed and split | Record | Field names; split before and after | priority_label; test to test | Fields exist in the schema for that version |
| before_hash and after_hash | Record | SHA-256 of the canonical record; null where not applicable | Two 64-character hex strings | after_hash matches the record in the new files |
| reason_code and evidence_ref | Record | taxonomy_merge, annotation_error, deletion_request, rights_withdrawn, duplicate | taxonomy_merge, review batch 14 | Removals and relabels must carry a reason |
| comparability | Record | none, rescore_required, retrain_recommended | rescore_required | Any relabel in an evaluation split is rescore_required |
| version, previous_version, metadata_version | Release | Semantic version strings | 2.0.0, 1.4.0, 3 | previous_version resolves; bump matches the strongest change type |
| counts and version_notes | Release | Counts per change type and per task family; text on compatibility | relabeled 312, split_moved 40 | Counts equal the record entries exactly |
| files and yanked | Release | Relative paths with SHA-256 per file; boolean | data/test-00001.parquet, false | Every file hash matches; yanked versions stay listed |
Where do the release-level fields come from in existing standards?
Most release-level fields already exist in published vocabularies, so a team can map them rather than invent new names. DCAT 3 adds a previous version property for building a version chain of a dataset's releases and a version notes property, defined as a description of changes between this version and the previous one, which should describe any backward compatibility issues.
Croissant separates the version of the dataset content from sdVersion, the version of its metadata, and strongly recommends recording a SHA-256 checksum for every file object in a versioned dataset so downloads can be checked against the declaration. The Data Package standard has a version property that should follow Semantic Versioning and its version recipe.
None of these vocabularies has a field for the tasks a change affects, and that is the gap the record-level file fills. Keep the task index in the package, give the changelog file a path relative to the package root and list it in the file manifest with its own checksum.
Worked example (illustrative): a support triage task set moves from 1.4.0 to 2.0.0
The numbers here are invented to show the format. A supplier's support-ticket triage set merges two priority categories after the source company changed its taxonomy. Reviewers relabel 312 records, 41 of them evaluation items, and move 40 near-duplicates from test to train to stop leakage. They also add 2,000 new records in a new split.
Taken alone, the additions would be a MINOR release. The relabels and split moves make it MAJOR, so the release is 2.0.0 with previous_version 1.4.0. The record file has 2,352 entries. The release counts read added 2,000, relabeled 312, split_moved 40, and the version notes state that scores on the old evaluation split are not comparable for the 41 relabeled items and the 40 moved items.
- Example record entry, written out: change_id chg-2.0.0-00041; record_id rec-88213; task_ids triage-eval-0907; change_type relabeled; fields_changed priority_label; reason_code taxonomy_merge; comparability rescore_required.
- Example split entry: record_id rec-90017; change_type split_moved; split test to train; reason_code duplicate; comparability rescore_required.
- What the buyer does: rescore the 81 affected evaluation items on the old and new labels, report both, and drop cached features for every record with a changed after_hash.
Can a buyer replay the change file and rebuild the release offline?
Yes, if the package carries the previous manifest, the new manifest, the task index and the record-level change file, all with relative paths and checksums. A validator can then replay the changes against the previous version and confirm it produces the new one, which proves the changelog is complete rather than a summary someone typed.
The exception is a live dataset. Croissant describes datasets that are refreshed continuously, recommends against checksums on files still being updated, and recommends bumping the version only for structural or backward-incompatible changes, not for appended data. For these, write a change file per closed partition and validate partitions once they stop changing; correcting a closed partition still needs a version bump, because history was rewritten.
- Every record_id in the change file exists in the old manifest, the new manifest or both, as its change type implies.
- Applying the entries to the previous version reproduces every after_hash in the new version.
- Release counts equal the record entries, grouped by change type and task family.
- The version bump matches the strongest change type present.
- Every file listed in the release entry matches its SHA-256 hash.
- No task ID in the change file is missing from the task index.
When should a change entry be held back or reviewed by a specialist?
Hold back detail when the change itself reveals something that should stay private, and get a specialist review when the change has legal or evaluation consequences beyond the dataset. A change file is shared with everyone who gets the package, so it must not become a side channel for content that was removed.
- Removals for deletion requests or withdrawn rights: record the record_id, reason code and before_hash only, never before values; ask privacy counsel whether even an unsalted hash of personal data is acceptable.
- Relabels that touch a published benchmark split: the evaluation owner should decide whether to publish a new split rather than edit the old one.
- Changes driven by a license amendment: the agreement owner confirms the reason code and the effective date.
- Large relabel waves from a guideline change: have the annotation lead document the guideline version, since one reason code can hide several decisions.
- Any release where the counts and the record file disagree: do not ship until they match.
Asking for this change format when you source data
Buyers get the most from this format when it is agreed before the first delivery, alongside the schema and the delivery specification, because retrofitting stable IDs and task indexes onto a delivered set is slow. Put the field list and the validator rules in the request so a supplier can say early whether its source systems can support them.
Because SourceX finds data at U.S. businesses only once a buyer asks, rather than keeping a catalog, the change-file fields and validator rules above can go straight into the request alongside scope, volume, language, format, timing and licensing needs.
Frequently asked questions
What should a record-level and release-level change format include?
At the record level: a stable record ID, the task IDs it feeds, a change type, the fields and split affected, before and after hashes, a reason code and a comparability flag. At the release level: the version, previous version, metadata version, counts per change type and task family, version notes, per-file checksums and a yanked flag. The release counts should be computable from the record entries.
Is correcting a label a patch or a major version?
For training and evaluation datasets, treat it as major. Croissant classifies edits to existing data as MAJOR, while the Data Package recipe calls error corrections a PATCH. A corrected label changes the expected output of a task, so results on the two versions are not directly comparable.
Can I keep using Keep a Changelog headings for datasets?
Yes, for the human summary of each release. Added, Changed, Removed and Fixed map well to release notes, and the YANKED tag is useful for pulled versions. Add a separate structured record-level file for pipelines, because the format itself is meant for people.
How do live or continuously refreshed datasets fit this format?
Croissant recommends not bumping the version for appended data in a live dataset and not checksumming files still being updated. Write change entries per closed partition and bump the version when a closed partition is corrected, since that rewrites history.
Should the changelog include the old values of removed records?
No, especially for deletion requests or withdrawn rights. Record the record ID, reason code and a before hash so the removal can be verified, and ask privacy counsel whether a hash of personal data is acceptable or should be salted or omitted.
Sources
- Keep a Changelog 1.1.0 says changelogs are for humans, not machines, defines the Added, Changed, Deprecated, Removed, Fixed and Security groups and the YANKED tag. Source
- Semantic Versioning 2.0.0 defines MAJOR.MINOR.PATCH and says the contents of a released version must not be modified. Source
- Croissant 1.1 recommends semantic versions for datasets, defines MAJOR as edited, removed or re-split data, separates content and metadata versions, recommends per-file SHA-256 checksums and describes live datasets. Source
- DCAT 3 defines previous version for a version chain and version notes describing changes and backward compatibility issues. Source
- The Data Package version recipe classifies error corrections as PATCH, added data as MINOR and field additions, removals or reordering as MAJOR. Source
- The Data Package standard's version property should follow Semantic Versioning and the Data Package Version recipe. Source
- Datasheets for Datasets asks whether a dataset has an erratum, will be updated to correct labeling errors or add and delete instances, and how obsolescence of older versions is communicated. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.