Systems and records
Dataset version release notes that tell buyers what to retest
By SourceX Editorial · Updated
9 sections
Short answer
A dataset change notice should tell buyers which version replaces which, what changed at record and field level and why, which delivered copies and splits are affected, and exactly what to rerun: which evaluations, whether any training, and which checks show the fix landed. It should end with the old version's status, a deadline and a way for each recipient to confirm receipt.
License terms
Dataset version release
Key takeaways
- Every change entry should end in a retest line that names the check, the split and the condition for stopping.
- Frictionless treats a value correction as a PATCH while Croissant treats it as MAJOR, so describe the effect in words rather than relying on the number.
- Trace affected records forward from the faulty source input to splits, versions and recipients, not backward from the complaint.
- A fix that touches a test split makes old and new scores incomparable; say so in its own line.
- Withdraw instead of upgrade when the old version contains data that should never have been delivered, and keep the YANKED entry in the changelog.
What should a dataset change notice tell buyers about retesting?
A dataset change notice should tell each buyer five things: which version replaces which, what changed at the record and field level, why it changed, which delivered copies and downstream uses are affected, and exactly what to rerun. The last item is the one most notes leave out. A buyer who reads that labels were corrected still has to guess whether a benchmark score, a fine-tuned checkpoint or a production filter is now wrong.
Treat the note as an instruction, not an announcement. Every change entry should end in a retest line that names the check, the split it applies to and the condition that means the buyer can stop. If nothing needs rerunning, say so and say why, because silence reads as uncertainty and buyers will either over-react by retraining everything or under-react by ignoring the notice.
Two documents do this job together. The changelog is the running history of every version. The change notice is the targeted message sent to the people who hold an affected version, with their copy, their deadline and their acknowledgment. This guide gives a template that serves both, so the notice is a filtered view of the changelog rather than a separate piece of writing that can drift from it.
Why the version number cannot carry the retest decision
A version number signals the size of a change, but the major standards disagree on how to classify the most common dataset change, a correction to existing values. The Data Package Version recipe from the Frictionless Data project tells publishers to increment only the PATCH number when they fix a data entry error by modifying a value. The Croissant format from MLCommons says a MAJOR increment is due when existing data has been edited, removed or shuffled across splits, and keeps PATCH for changes in serialization or packaging where the data stays the same.
So the same correction can arrive as 2.3.1 from one supplier and 3.0.0 from another. A buyer who automates retests from the version number alone will either miss a relabel that changes evaluation results or rerun pipelines after a harmless file format change. The fix is to write the effect on the data in the note itself and treat the number as a summary.
One rule from Semantic Versioning does transfer cleanly: once a version is released, its contents must not be modified, and any modification is released as a new version. Apply it to data without exception. A corrected file uploaded under the old version ID breaks every comparison a buyer has made, because their stored scores now point at bytes that no longer exist.
| Change to the data | Frictionless recipe | Croissant guidance | What the note must state |
|---|---|---|---|
| Correct wrong values in existing records | PATCH | MAJOR (existing data edited) | Record IDs touched, fields touched, split membership, whether evaluation scores are comparable across versions |
| Append new records | MINOR | MINOR if old record sets remain retrievable | Count added, date range, which split received them, whether existing splits changed |
| Add, remove or reorder fields | MAJOR | MAJOR if old data is no longer easily accessible as it was | Field names, types, defaults for older records, the parser change a buyer needs |
| Re-serialize or repackage with the same data | No version change for relocating data; descriptive metadata edits are PATCH | PATCH | New checksums and a statement that values are unchanged, so no model retest is needed |
| Remove records | MAJOR when it deletes data referenced by another resource | MAJOR | Record IDs removed, reason class, whether buyers must delete local copies |
What should the dataset release note template include?
The template below has ten fields. Fields 1 to 3 identify the release and the problem, 4 to 6 describe the change and its proof, and 7 to 10 turn it into an instruction for each recipient. Keep it as structured data next to the files and render the human-readable note from it, so the two cannot disagree.
The change classes in field 3 borrow the six types from Keep a Changelog (Added, Changed, Deprecated, Removed, Fixed and Security) and its convention of flagging a pulled release as YANKED. For data, add Withdrawn for records removed because the right to use them ended, since that class carries obligations the others do not.
| Field | What to write | Illustrative entry | Why the buyer needs it |
|---|---|---|---|
| 1. Release ID and parent | New version, the version it replaces, release date, immutable reference such as a tag or commit | support-tickets 3.0.0, parent 2.3.0, tag v3.0.0 | Lets buyers pin the exact revision their runs used and diff against it |
| 2. Issue ID and how it was found | Tracker reference, reporter type (internal QA, buyer report, source owner), date found | DQ-118, buyer report, found 2026-10-02 | Shows whether the problem was caught upstream or by someone already relying on the data |
| 3. Severity and change class | Severity: cosmetic, quality, evaluation-affecting, rights or privacy. Class: Added, Changed, Deprecated, Removed, Fixed, Security, Withdrawn | Evaluation-affecting; Fixed | Sets how fast buyers must act and lets them filter the changelog to what their pipelines use |
| 4. Affected scope | Record ID list or query, counts per split, fields touched, source systems | Illustrative: 412 records, 380 train and 32 test, field refund_reason | Turns a vague fix into something a buyer can join against their own copy |
| 5. Root cause and repair | What went wrong, the upstream record or rule it traces to, and the exact rule or review used to correct it | Mapping table merged two reason codes after a CRM picklist change; re-mapped from picklist history with second-annotator review | Lets buyers judge whether similar errors remain and audit the fix instead of trusting it |
| 6. Regression checks run | Checks rerun before release and their pass condition | Label distribution diff, duplicate scan, de-identification scan, row count reconciliation | Shows the fix did not break something else |
| 7. Retest instruction | What each buyer should rerun, on which split, and when they can stop | Rescore test split; retraining optional unless refund_reason is a training target | The core of the notice; without it buyers guess |
| 8. Recipient and version log | Every recipient holding an affected version, delivery date, delivery method | Three recipients hold 2.3.0; one holds 2.2.1 | Defines who must receive the notice and which diff each one needs |
| 9. Old version status and deadline | Usable, deprecated, or withdrawn; date by which to switch or delete | 2.3.0 deprecated; switch by 2026-11-15 | Separates a recommended upgrade from an obligation |
| 10. Integrity and acknowledgment | Manifest and per-file checksums, how recipients confirm receipt and action taken | SHA-256 per file; recipient returns checksum of received manifest | Proves each buyer has the corrected bytes and closes the loop |
Tracing a source fault forward to affected records and recipients
Answer it by walking the lineage forward from the source record, not backward from the complaint. The complaint names the records someone noticed; the lineage tells you every record built from the same faulty input, every split it landed in and every recipient who received those splits.
This only works if three links were recorded at build time: source record to derived record ID, derived record ID to split and version, and version to recipient. Croissant recommends recording a sha256 checksum for every file in a versioned dataset so a downloaded copy can be checked against its declaration, and Hugging Face lets a consumer load a dataset at a specific git tag or commit through the revision parameter. Both make the version-to-copy link checkable. The source-to-derived link is yours to keep.
- Start from the faulty input: the source system, table and field, the rule or mapping that misbehaved, and the date range it was active.
- Query derived records whose lineage includes that input. Record the count and the ID list, not a sample.
- Group the affected IDs by split. A fix that touches a test split changes reported scores; a fix confined to training data changes what a model learned.
- Join the IDs to every released version that contains them. Older versions often carry the same error under a different version ID.
- Join versions to the recipient log. Each recipient gets a notice scoped to the versions they actually hold.
- Check derived artifacts you produced yourself, such as summary statistics, sample files or dataset cards, and correct them in the same release.
Which retest instruction fits each kind of change?
Pick the retest line by asking what the buyer has built on top of the data, then match the change to that use. The table gives defaults; a buyer's contract or internal policy can make them stricter.
| Change | Typical severity | Retest instruction to write | Old version status |
|---|---|---|---|
| Labels or values corrected in a test or evaluation split | Evaluation-affecting | Rescore every model evaluated on the old split; mark old and new scores as not comparable | Deprecated with a switch date |
| Labels or values corrected in training data only | Quality | Re-run the evaluation suite on models trained on the old version; retrain only if the corrected field is a target or a filter | Deprecated |
| Records appended to a new split or period | Cosmetic for existing users | No retest needed for existing work; new records can be evaluated separately | Usable |
| Field renamed, retyped or removed | Quality | Rerun ingestion and schema tests before any model work; check feature code that reads the field | Usable until the stated deprecation date |
| Records withdrawn because usage rights ended | Rights or privacy | Delete the listed records from local copies and derived sets as the agreement requires; confirm in writing | Withdrawn |
| Personal data found that should have been removed | Rights or privacy | Stop using the affected version at once, purge it and any derived copies, then load the replacement | Withdrawn and flagged YANKED |
| Re-serialization with identical values | Cosmetic | Verify new checksums; no model retest | Usable |
A worked example: relabeling refund reasons in a support ticket set
This case is illustrative; the dataset, counts and dates are invented to show the template in use. A supplier delivers a support ticket dataset with a refund_reason label derived from a CRM picklist. A buyer reports that tickets about duplicate billing are labeled as product defects. The supplier traces the cause to a mapping table that merged two picklist values after the CRM administrator renamed one of them.
At a glance
A worked example: relabeling refund reasons in a support ticket set
Walking the lineage forward finds 412 affected tickets, about 1.6 percent of the set: 380 in the training split and 32 in the test split. Versions 2.2.1 and 2.3.0 both contain them; three recipients hold 2.3.0 and one still holds 2.2.1. The supplier re-maps the labels from the picklist history, has a second reviewer check the 27 tickets where the history is ambiguous, and reruns the label distribution diff, the duplicate scan and the de-identification scan before release.
The exception is the test split. Because 32 test labels changed, any accuracy number computed on the old test split is no longer comparable with one computed on the new split, even for the same model. Under the Croissant rule this is a MAJOR change; under the Frictionless recipe it is a PATCH. The supplier releases 3.0.0, says in the note which rule it followed, and writes the comparability warning as its own line so no buyer misses it.
The resulting notice to the three holders of 2.3.0 reads, in structured form:
- Release: 3.0.0 replaces 2.3.0; tag v3.0.0; manifest checksum listed in the attached file.
- Issue: DQ-118, found by a buyer report; severity evaluation-affecting; class Fixed.
- Scope: 412 tickets, field refund_reason; 380 train, 32 test; ID list attached.
- Cause and repair: picklist rename merged two reason codes; labels re-mapped from picklist history; 27 ambiguous tickets reviewed by a second annotator.
- Retest: rescore models on the new test split and do not compare with old scores; retrain only if refund_reason is a training target or a sampling filter.
- Status: 2.3.0 deprecated, switch by 2026-11-15; reply with the checksum of the received manifest.
- The holder of 2.2.1 receives the same note with a diff from 2.2.1, because one other change landed in between.
When the old version has to be pulled instead of patched
A note with an upgrade path is enough when the old version is merely worse. It is not enough when the old version should not be used at all: when it contains personal data that was supposed to be removed, records the supplier no longer has the right to license, or content that breaks the agreement's scope. In those cases the old version is withdrawn, the notice says stop rather than switch, and the deadline is immediate.
Keep a Changelog's advice for pulled software releases applies here: list the withdrawn version in the changelog with a loud YANKED marker instead of quietly deleting it, so later readers know it existed and why it must not be used. Delete the files, keep the record.
Escalate before writing the notice whenever the cause touches privacy law, a data subject request or the end of a usage right. Those notices can create obligations for buyers that only the agreement defines, and the wording should be checked by whoever owns the contract and, where personal data is involved, by privacy counsel.
Stop and narrow the scope when the lineage walk cannot be finished. If you cannot say which records are affected, withdraw the smallest version range you can prove is clean of the cause, tell buyers the scope is provisional, and send a second notice when the trace is complete. A provisional notice beats a precise one that arrives after the buyer has shipped.
What to record before the first delivery so later notices are cheap
Most of the cost of a change notice is paid at build time or not at all. The Datasheets for Datasets questionnaire asks dataset creators whether the data will be updated, how often, by whom and how updates will be communicated, and whether an erratum exists. Answer those questions before the first delivery, and the notice process already has an owner and a channel when the first fix is needed. Croissant's responsible AI vocabulary has a matching property, dataReleaseMaintenancePlan, for the update timeframe, the maintainers and the deprecation policy.
Name one owner for change notices and one channel per recipient before anything ships. A notice sent to the engineer who downloaded the files three months ago, who has since moved teams, is a notice nobody acts on. Ask each buyer at delivery for a role-based contact for data issues and record it next to the version log, and agree the response you expect: a reply with the checksum of the received manifest and a short statement of what was rerun.
Decide in advance how long superseded versions stay available. Buyers who need to reproduce an old result want the old bytes; buyers bound by a withdrawal must not keep them. Write both cases into the maintenance plan so the notice can point to a rule instead of a negotiation.
Record at build time: stable record IDs, the lineage from each derived record to its source, split assignment, per-file checksums, the recipient and version log, and the maintenance plan itself. With those in place, the ten fields above take an hour to fill rather than a week of reconstruction.
How change notices fit a records licensing agreement
For a business licensing its own records, change notices are part of what the agreement should anticipate. With SourceX, a company approves the scope and terms and signs before any data leaves its hands, and after that the de-identified dataset can be used, sublicensed and distributed only as the agreement allows, so a correction or withdrawal has to travel to everyone who holds the affected version. De-identification is done by SourceX through approved processing routes, which makes the de-identification scan in field 6 a shared check rather than one side's promise.
Large releases are not hosted by SourceX; they stay in the seller's storage or move on encrypted drives, so per-file checksums are how both sides confirm which bytes a notice refers to. A supplier unsure whether its release history would qualify can find out through a short assessment, before any file is shared.
Frequently asked questions
What is the difference between a dataset changelog and a change notice?
The changelog is the complete, public history of every version and what changed in it. A change notice is a targeted message to the people who hold an affected version, with their specific diff, retest instruction and deadline. Generate the notice from the changelog entry so the two never disagree.
Should a correction to existing records be a patch or a major version?
The standards differ. The Frictionless Data Package Version recipe treats fixing a data entry error as a PATCH, while Croissant treats any edit to existing data as a MAJOR change. Pick one rule, state it in your notes, and always describe the effect on scores and splits in words, because buyers cannot infer it from the number.
Do buyers need to retrain after every dataset fix?
No. Retraining is warranted when the corrected field is a training target or a filter that shaped the training sample, or when the fix removes data that may no longer be used. Corrections confined to an evaluation split usually require rescoring, not retraining, and appended records need no retest of existing work.
How do you know which recipients to notify?
Keep a recipient and version log from the first delivery: who received which version, when and how. Join the affected record IDs to versions, and versions to that log. Every holder of an affected version gets a notice scoped to the version they hold, including holders of older versions that carry the same error.
Can you replace files inside an existing version instead of releasing a new one?
You should not. Semantic Versioning's rule that released contents must not be modified applies equally to data: a buyer's stored scores and checksums refer to the old bytes, and silently replacing them breaks every comparison. Release a new version and deprecate or withdraw the old one.
Sources
- Semantic Versioning 2.0.0 states that once a versioned package is released its contents must not be modified, and any modification must be released as a new version. Source
- Keep a Changelog 1.1.0 groups changes as Added, Changed, Deprecated, Removed, Fixed and Security, and recommends listing pulled releases with a [YANKED] tag rather than omitting them. Source
- The Data Package standard defines a version property that should conform to Semantic Versioning and follow the Data Package Version recipe. Source
- The Data Package Version recipe says to increment PATCH when fixing a data entry error by modifying a value, MINOR when appending data, and MAJOR when adding, removing or reordering fields. Source
- The Croissant specification recommends MAJOR.MINOR.PATCH versioning in which edited, removed or reshuffled existing data is a MAJOR change and PATCH means the data is unchanged, and strongly recommends sha256 checksums for every file in versioned datasets. Source
- The Croissant RAI vocabulary includes dataReleaseMaintenancePlan for versioning information such as the updating timeframe, the maintainers and the deprecation policies. Source
- The Hugging Face datasets library loads a specific dataset version through the revision parameter, which accepts a tag, branch or commit hash. Source
- Datasheets for Datasets asks creators whether the dataset will be updated, how often, by whom and how updates will be communicated, and whether there is an erratum. Source
See if your company qualifies
A short company assessment. No data uploads are needed.