Skip to content

Systems and records

Dataset version release notes that tell buyers what to retest

By SourceX Editorial · Updated

9 sections

Short answer

A dataset change notice should tell buyers which version replaces which, what changed at record and field level and why, which delivered copies and splits are affected, and exactly what to rerun: which evaluations, whether any training, and which checks show the fix landed. It should end with the old version's status, a deadline and a way for each recipient to confirm receipt.

License terms

Dataset version release

This page covers dataset version release notes that tell buyers what to retest.Illustrative

Key takeaways

  • Every change entry should end in a retest line that names the check, the split and the condition for stopping.
  • Frictionless treats a value correction as a PATCH while Croissant treats it as MAJOR, so describe the effect in words rather than relying on the number.
  • Trace affected records forward from the faulty source input to splits, versions and recipients, not backward from the complaint.
  • A fix that touches a test split makes old and new scores incomparable; say so in its own line.
  • Withdraw instead of upgrade when the old version contains data that should never have been delivered, and keep the YANKED entry in the changelog.

What should a dataset change notice tell buyers about retesting?

A dataset change notice should tell each buyer five things: which version replaces which, what changed at the record and field level, why it changed, which delivered copies and downstream uses are affected, and exactly what to rerun. The last item is the one most notes leave out. A buyer who reads that labels were corrected still has to guess whether a benchmark score, a fine-tuned checkpoint or a production filter is now wrong.

Treat the note as an instruction, not an announcement. Every change entry should end in a retest line that names the check, the split it applies to and the condition that means the buyer can stop. If nothing needs rerunning, say so and say why, because silence reads as uncertainty and buyers will either over-react by retraining everything or under-react by ignoring the notice.

Two documents do this job together. The changelog is the running history of every version. The change notice is the targeted message sent to the people who hold an affected version, with their copy, their deadline and their acknowledgment. This guide gives a template that serves both, so the notice is a filtered view of the changelog rather than a separate piece of writing that can drift from it.

Why the version number cannot carry the retest decision

A version number signals the size of a change, but the major standards disagree on how to classify the most common dataset change, a correction to existing values. The Data Package Version recipe from the Frictionless Data project tells publishers to increment only the PATCH number when they fix a data entry error by modifying a value. The Croissant format from MLCommons says a MAJOR increment is due when existing data has been edited, removed or shuffled across splits, and keeps PATCH for changes in serialization or packaging where the data stays the same.

So the same correction can arrive as 2.3.1 from one supplier and 3.0.0 from another. A buyer who automates retests from the version number alone will either miss a relabel that changes evaluation results or rerun pipelines after a harmless file format change. The fix is to write the effect on the data in the note itself and treat the number as a summary.

One rule from Semantic Versioning does transfer cleanly: once a version is released, its contents must not be modified, and any modification is released as a new version. Apply it to data without exception. A corrected file uploaded under the old version ID breaks every comparison a buyer has made, because their stored scores now point at bytes that no longer exist.

Why the version number cannot carry the retest decision
Change to the dataFrictionless recipeCroissant guidanceWhat the note must state
Correct wrong values in existing recordsPATCHMAJOR (existing data edited)Record IDs touched, fields touched, split membership, whether evaluation scores are comparable across versions
Append new recordsMINORMINOR if old record sets remain retrievableCount added, date range, which split received them, whether existing splits changed
Add, remove or reorder fieldsMAJORMAJOR if old data is no longer easily accessible as it wasField names, types, defaults for older records, the parser change a buyer needs
Re-serialize or repackage with the same dataNo version change for relocating data; descriptive metadata edits are PATCHPATCHNew checksums and a statement that values are unchanged, so no model retest is needed
Remove recordsMAJOR when it deletes data referenced by another resourceMAJORRecord IDs removed, reason class, whether buyers must delete local copies

What should the dataset release note template include?

The template below has ten fields. Fields 1 to 3 identify the release and the problem, 4 to 6 describe the change and its proof, and 7 to 10 turn it into an instruction for each recipient. Keep it as structured data next to the files and render the human-readable note from it, so the two cannot disagree.

The change classes in field 3 borrow the six types from Keep a Changelog (Added, Changed, Deprecated, Removed, Fixed and Security) and its convention of flagging a pulled release as YANKED. For data, add Withdrawn for records removed because the right to use them ended, since that class carries obligations the others do not.

What should the dataset release note template include?
FieldWhat to writeIllustrative entryWhy the buyer needs it
1. Release ID and parentNew version, the version it replaces, release date, immutable reference such as a tag or commitsupport-tickets 3.0.0, parent 2.3.0, tag v3.0.0Lets buyers pin the exact revision their runs used and diff against it
2. Issue ID and how it was foundTracker reference, reporter type (internal QA, buyer report, source owner), date foundDQ-118, buyer report, found 2026-10-02Shows whether the problem was caught upstream or by someone already relying on the data
3. Severity and change classSeverity: cosmetic, quality, evaluation-affecting, rights or privacy. Class: Added, Changed, Deprecated, Removed, Fixed, Security, WithdrawnEvaluation-affecting; FixedSets how fast buyers must act and lets them filter the changelog to what their pipelines use
4. Affected scopeRecord ID list or query, counts per split, fields touched, source systemsIllustrative: 412 records, 380 train and 32 test, field refund_reasonTurns a vague fix into something a buyer can join against their own copy
5. Root cause and repairWhat went wrong, the upstream record or rule it traces to, and the exact rule or review used to correct itMapping table merged two reason codes after a CRM picklist change; re-mapped from picklist history with second-annotator reviewLets buyers judge whether similar errors remain and audit the fix instead of trusting it
6. Regression checks runChecks rerun before release and their pass conditionLabel distribution diff, duplicate scan, de-identification scan, row count reconciliationShows the fix did not break something else
7. Retest instructionWhat each buyer should rerun, on which split, and when they can stopRescore test split; retraining optional unless refund_reason is a training targetThe core of the notice; without it buyers guess
8. Recipient and version logEvery recipient holding an affected version, delivery date, delivery methodThree recipients hold 2.3.0; one holds 2.2.1Defines who must receive the notice and which diff each one needs
9. Old version status and deadlineUsable, deprecated, or withdrawn; date by which to switch or delete2.3.0 deprecated; switch by 2026-11-15Separates a recommended upgrade from an obligation
10. Integrity and acknowledgmentManifest and per-file checksums, how recipients confirm receipt and action takenSHA-256 per file; recipient returns checksum of received manifestProves each buyer has the corrected bytes and closes the loop

Tracing a source fault forward to affected records and recipients

Answer it by walking the lineage forward from the source record, not backward from the complaint. The complaint names the records someone noticed; the lineage tells you every record built from the same faulty input, every split it landed in and every recipient who received those splits.

This only works if three links were recorded at build time: source record to derived record ID, derived record ID to split and version, and version to recipient. Croissant recommends recording a sha256 checksum for every file in a versioned dataset so a downloaded copy can be checked against its declaration, and Hugging Face lets a consumer load a dataset at a specific git tag or commit through the revision parameter. Both make the version-to-copy link checkable. The source-to-derived link is yours to keep.

  • Start from the faulty input: the source system, table and field, the rule or mapping that misbehaved, and the date range it was active.
  • Query derived records whose lineage includes that input. Record the count and the ID list, not a sample.
  • Group the affected IDs by split. A fix that touches a test split changes reported scores; a fix confined to training data changes what a model learned.
  • Join the IDs to every released version that contains them. Older versions often carry the same error under a different version ID.
  • Join versions to the recipient log. Each recipient gets a notice scoped to the versions they actually hold.
  • Check derived artifacts you produced yourself, such as summary statistics, sample files or dataset cards, and correct them in the same release.

Which retest instruction fits each kind of change?

Pick the retest line by asking what the buyer has built on top of the data, then match the change to that use. The table gives defaults; a buyer's contract or internal policy can make them stricter.

Which retest instruction fits each kind of change?
ChangeTypical severityRetest instruction to writeOld version status
Labels or values corrected in a test or evaluation splitEvaluation-affectingRescore every model evaluated on the old split; mark old and new scores as not comparableDeprecated with a switch date
Labels or values corrected in training data onlyQualityRe-run the evaluation suite on models trained on the old version; retrain only if the corrected field is a target or a filterDeprecated
Records appended to a new split or periodCosmetic for existing usersNo retest needed for existing work; new records can be evaluated separatelyUsable
Field renamed, retyped or removedQualityRerun ingestion and schema tests before any model work; check feature code that reads the fieldUsable until the stated deprecation date
Records withdrawn because usage rights endedRights or privacyDelete the listed records from local copies and derived sets as the agreement requires; confirm in writingWithdrawn
Personal data found that should have been removedRights or privacyStop using the affected version at once, purge it and any derived copies, then load the replacementWithdrawn and flagged YANKED
Re-serialization with identical valuesCosmeticVerify new checksums; no model retestUsable

A worked example: relabeling refund reasons in a support ticket set

This case is illustrative; the dataset, counts and dates are invented to show the template in use. A supplier delivers a support ticket dataset with a refund_reason label derived from a CRM picklist. A buyer reports that tickets about duplicate billing are labeled as product defects. The supplier traces the cause to a mapping table that merged two picklist values after the CRM administrator renamed one of them.

At a glance

A worked example: relabeling refund reasons in a support ticket set

Walking the lineage forward finds 412 affected tickets, about 1.6 percent of the set: 380 in the training split and 32 in the test split. Versions 2.2.1 and 2.3.0 both contain them; three recipients hold 2.3.0 and one still holds 2.2.1. The supplier re-maps the labels from the picklist history, has a second reviewer check the 27 tickets where the history is ambiguous, and reruns the label distribution diff, the duplicate scan and the de-identification scan before release.

The exception is the test split. Because 32 test labels changed, any accuracy number computed on the old test split is no longer comparable with one computed on the new split, even for the same model. Under the Croissant rule this is a MAJOR change; under the Frictionless recipe it is a PATCH. The supplier releases 3.0.0, says in the note which rule it followed, and writes the comparability warning as its own line so no buyer misses it.

The resulting notice to the three holders of 2.3.0 reads, in structured form:

  • Release: 3.0.0 replaces 2.3.0; tag v3.0.0; manifest checksum listed in the attached file.
  • Issue: DQ-118, found by a buyer report; severity evaluation-affecting; class Fixed.
  • Scope: 412 tickets, field refund_reason; 380 train, 32 test; ID list attached.
  • Cause and repair: picklist rename merged two reason codes; labels re-mapped from picklist history; 27 ambiguous tickets reviewed by a second annotator.
  • Retest: rescore models on the new test split and do not compare with old scores; retrain only if refund_reason is a training target or a sampling filter.
  • Status: 2.3.0 deprecated, switch by 2026-11-15; reply with the checksum of the received manifest.
  • The holder of 2.2.1 receives the same note with a diff from 2.2.1, because one other change landed in between.

When the old version has to be pulled instead of patched

A note with an upgrade path is enough when the old version is merely worse. It is not enough when the old version should not be used at all: when it contains personal data that was supposed to be removed, records the supplier no longer has the right to license, or content that breaks the agreement's scope. In those cases the old version is withdrawn, the notice says stop rather than switch, and the deadline is immediate.

Keep a Changelog's advice for pulled software releases applies here: list the withdrawn version in the changelog with a loud YANKED marker instead of quietly deleting it, so later readers know it existed and why it must not be used. Delete the files, keep the record.

Escalate before writing the notice whenever the cause touches privacy law, a data subject request or the end of a usage right. Those notices can create obligations for buyers that only the agreement defines, and the wording should be checked by whoever owns the contract and, where personal data is involved, by privacy counsel.

Stop and narrow the scope when the lineage walk cannot be finished. If you cannot say which records are affected, withdraw the smallest version range you can prove is clean of the cause, tell buyers the scope is provisional, and send a second notice when the trace is complete. A provisional notice beats a precise one that arrives after the buyer has shipped.

What to record before the first delivery so later notices are cheap

Most of the cost of a change notice is paid at build time or not at all. The Datasheets for Datasets questionnaire asks dataset creators whether the data will be updated, how often, by whom and how updates will be communicated, and whether an erratum exists. Answer those questions before the first delivery, and the notice process already has an owner and a channel when the first fix is needed. Croissant's responsible AI vocabulary has a matching property, dataReleaseMaintenancePlan, for the update timeframe, the maintainers and the deprecation policy.

Name one owner for change notices and one channel per recipient before anything ships. A notice sent to the engineer who downloaded the files three months ago, who has since moved teams, is a notice nobody acts on. Ask each buyer at delivery for a role-based contact for data issues and record it next to the version log, and agree the response you expect: a reply with the checksum of the received manifest and a short statement of what was rerun.

Decide in advance how long superseded versions stay available. Buyers who need to reproduce an old result want the old bytes; buyers bound by a withdrawal must not keep them. Write both cases into the maintenance plan so the notice can point to a rule instead of a negotiation.

Record at build time: stable record IDs, the lineage from each derived record to its source, split assignment, per-file checksums, the recipient and version log, and the maintenance plan itself. With those in place, the ten fields above take an hour to fill rather than a week of reconstruction.

How change notices fit a records licensing agreement

For a business licensing its own records, change notices are part of what the agreement should anticipate. With SourceX, a company approves the scope and terms and signs before any data leaves its hands, and after that the de-identified dataset can be used, sublicensed and distributed only as the agreement allows, so a correction or withdrawal has to travel to everyone who holds the affected version. De-identification is done by SourceX through approved processing routes, which makes the de-identification scan in field 6 a shared check rather than one side's promise.

Large releases are not hosted by SourceX; they stay in the seller's storage or move on encrypted drives, so per-file checksums are how both sides confirm which bytes a notice refers to. A supplier unsure whether its release history would qualify can find out through a short assessment, before any file is shared.

Frequently asked questions

What is the difference between a dataset changelog and a change notice?

The changelog is the complete, public history of every version and what changed in it. A change notice is a targeted message to the people who hold an affected version, with their specific diff, retest instruction and deadline. Generate the notice from the changelog entry so the two never disagree.

Should a correction to existing records be a patch or a major version?

The standards differ. The Frictionless Data Package Version recipe treats fixing a data entry error as a PATCH, while Croissant treats any edit to existing data as a MAJOR change. Pick one rule, state it in your notes, and always describe the effect on scores and splits in words, because buyers cannot infer it from the number.

Do buyers need to retrain after every dataset fix?

No. Retraining is warranted when the corrected field is a training target or a filter that shaped the training sample, or when the fix removes data that may no longer be used. Corrections confined to an evaluation split usually require rescoring, not retraining, and appended records need no retest of existing work.

How do you know which recipients to notify?

Keep a recipient and version log from the first delivery: who received which version, when and how. Join the affected record IDs to versions, and versions to that log. Every holder of an affected version gets a notice scoped to the version they hold, including holders of older versions that carry the same error.

Can you replace files inside an existing version instead of releasing a new one?

You should not. Semantic Versioning's rule that released contents must not be modified applies equally to data: a buyer's stored scores and checksums refer to the old bytes, and silently replacing them breaks every comparison. Release a new version and deprecate or withdraw the old one.

Sources

  • Semantic Versioning 2.0.0 states that once a versioned package is released its contents must not be modified, and any modification must be released as a new version. Source
  • Keep a Changelog 1.1.0 groups changes as Added, Changed, Deprecated, Removed, Fixed and Security, and recommends listing pulled releases with a [YANKED] tag rather than omitting them. Source
  • The Data Package standard defines a version property that should conform to Semantic Versioning and follow the Data Package Version recipe. Source
  • The Data Package Version recipe says to increment PATCH when fixing a data entry error by modifying a value, MINOR when appending data, and MAJOR when adding, removing or reordering fields. Source
  • The Croissant specification recommends MAJOR.MINOR.PATCH versioning in which edited, removed or reshuffled existing data is a MAJOR change and PATCH means the data is unchanged, and strongly recommends sha256 checksums for every file in versioned datasets. Source
  • The Croissant RAI vocabulary includes dataReleaseMaintenancePlan for versioning information such as the updating timeframe, the maintainers and the deprecation policies. Source
  • The Hugging Face datasets library loads a specific dataset version through the revision parameter, which accepts a tag, branch or commit hash. Source
  • Datasheets for Datasets asks creators whether the dataset will be updated, how often, by whom and how updates will be communicated, and whether there is an erratum. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies