Systems and records
Software release note provenance: proving the notes match what shipped
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
To verify that release notes describe what shipped, test each entry through a chain of records: the entry cites a merged change, the change is contained in the release tag, the tag is the source commit in the build provenance, the provenance subject digest matches the artifact, and that digest is what the deployment record shows in production. Then list every change in the tag range without an entry as an omission.
Provenance trail
Software release note
Key takeaways
- A release note entry is proven only when it links through change, tag, build provenance and artifact digest to the deployment record.
- Omissions matter as much as false entries: every change in the tag range needs an entry or a recorded exclusion reason.
- Generated notes depend on labels and exclusions, and hand-written changelogs are written at merge time, so both drift from the release.
- Semantic versioning and commit types give a mechanical consistency check that flags entries for review without proving them.
- Label each task's verification mode: rebuilt and matched, provenance verified, or neither.
- The benchmark scores per-entry verdicts, omission recall and evidence citations separately; no results are reported here.
How do you verify that release notes describe the software actually shipped?
Treat every release note entry as a claim and walk it through a chain of evidence to the running artifact. The entry must cite a change; the change must be contained in the tagged commit; the tagged commit must be the source recorded in the build's provenance; the provenance must name the exact artifact digest; and that digest must be what the deployment record says went to production. An entry that breaks any link is unproven.
The check also runs backwards. Every change between the previous release tag and this one should either appear in the notes or be deliberately excluded with a reason. A shipped change with no entry is an omission, and omissions are easy to miss because nothing on the page points to them: a dependency bump, a default that changed, a fix that was backported without a line in the notes.
This is the task an agent should be able to perform, and it is the task the benchmark below measures. The agent gets the notes and the records; it must return a verdict per entry, a list of omissions and the evidence for each, not a summary that sounds plausible.
Why the notes and the artifact drift apart
Release notes are written by people or generated from labels, and both routes lose information. GitHub documents that its automatically generated release notes list merged pull requests, contributors and a link to the full changelog, and that a release.yml file can sort pull requests into categories by label and exclude labels or authors. A mislabeled pull request lands in the wrong category; an excluded label silently drops a change. GitHub also tells users to check the generated notes before publishing.
Hand-written changelogs have different failure modes. Keep a Changelog, a published changelog convention, says changelogs are for humans and groups entries by type: Added, Changed, Deprecated, Removed, Fixed and Security, with an Unreleased section collecting changes before a release. Entries are written when the change merges, which may be weeks before a release is cut and may not survive a later revert.
The tag itself can mislead. GitHub notes that releases are based on Git tags and that a tag date may differ from the release date. Hotfixes built after the tag, artifacts rebuilt from a moved tag, and features merged but switched off by a flag all create a release whose notes are accurate about the code and wrong about what users received.
Which record proves each link in the chain?
Each link has a record that proves it and a check an agent or reviewer can run. Supply-chain specifications make the build links precise. SLSA build provenance records the build's external parameters, the resolved dependencies, such as the specific git commit a repository reference resolved to, and identifies the outputs as the subject. The in-toto Statement that carries it requires a digest for every subject and treats subjects as immutable.
| Link | Claim being tested | Evidence record | Check |
|---|---|---|---|
| Entry to change | The note describes a real change | Pull request or commit referenced by the entry | Referenced change exists, is merged and matches the described behavior |
| Change to tag | The change is in this release | Commit graph between previous and current tag | Merge or commit is an ancestor of the tag and not of the previous tag, and not reverted |
| Tag to build | The build used this source | Build provenance resolved dependencies | Recorded source commit equals the tagged commit |
| Build to artifact | This artifact came from that build | Provenance subject digest | Digest of the artifact equals the subject digest |
| Artifact to deployment | This artifact reached production | Deployment record or registry pull record | Deployed digest equals the subject digest, with time and environment |
| Deployment to behavior | Users received the change | Feature flag state and configuration at release time | Flag or setting enabling the change was on for the stated audience |
The release artifact evidence matching benchmark
Each benchmark task is one release. The agent receives the release note text, read access to the repository between the previous and current tags, the build provenance, the deployment record and a flag-state snapshot. It must label every note entry, list every omission and cite the evidence records it used. Gold labels come from two independent engineers who reconcile disagreements before the task is admitted.
The benchmark is a method; no scores or results are reported here. Score per-entry label accuracy, omission recall and evidence citation correctness separately, because an agent that labels well but cites the wrong commit has not verified anything.
| Verdict label | Definition | Required evidence | Typical cause |
|---|---|---|---|
| Supported | Every link from entry to deployment holds | Change id, ancestry, provenance commit, matching digests | Normal release |
| Type mismatch | Change shipped but the note misstates its kind or impact | Change content and its declared type | Breaking change described as a fix |
| Not in release | Described change is absent or reverted at the tag | Ancestry check or revert commit | Entry written at merge, change reverted later |
| Shipped but off | Code shipped but the enabling flag was off | Flag state at release time | Staged rollout or dark launch |
| Omitted | A change in the tag range has no entry and no exclusion reason | Commit or pull request in range | Excluded label, backport, dependency bump |
| Unverifiable | A link cannot be checked from the records supplied | Statement of which record is missing | No provenance, moved tag, rebuilt artifact |
Using version numbers and commit types as a consistency check
Version numbers and structured commit messages give the agent a second, independent signal about impact. Semantic Versioning says a patch increment is for backward compatible bug fixes, a minor increment for new backward compatible functionality and a major increment for backward incompatible changes. It also says a released version's contents must not be modified; any modification must ship as a new version.
Conventional Commits maps commit types onto those increments: fix correlates with a patch, feat with a minor version, and a BREAKING CHANGE footer or an exclamation mark after the type with a major version. Other types are allowed but carry no versioning effect unless they include a breaking change.
These conventions turn some mismatches into mechanical checks. A patch release whose range contains a breaking change footer has a type mismatch somewhere, either in the notes, the version or the commit message. A changed artifact digest under an existing version number is a direct conflict with the rule that released contents are not modified. Neither check proves the notes are right, but each one flags the entries a reviewer should open first.
A worked release with an exception
This release is illustrative. The project, version numbers, counts and identifiers are invented, and no real software is described.
At a glance
A worked release with an exception
Version 4.3.0 ships with five note entries. The tag range contains eleven merged pull requests. Provenance exists for the container image, and the deployment record shows one digest promoted to production two days after the tag.
- Entries one and two: supported. Each cites a merged pull request that is an ancestor of the tag; the provenance commit equals the tag; the deployed digest equals the provenance subject.
- Entry three, listed under Fixed: type mismatch. The underlying commit carries a BREAKING CHANGE footer that removes a configuration option, and the version is a minor increment.
- Entry four: not in release. The pull request merged, but a revert commit sits between it and the tag.
- Entry five, a new export screen: shipped but off. The code is present and the flag snapshot shows the feature disabled for all customers at release time.
- Omissions: of the eleven pull requests, six are covered by entries; four carry an exclusion label for internal tooling with a recorded reason; one dependency upgrade that changes a default timeout has no entry and no reason, so it is listed as omitted.
- Exception: a second image with the same version tag but a different digest appears in the registry a day later. No provenance names it and no deployment record shows it in production. The task records it as unverifiable for any claim about that image, rather than letting the matching digest for the first image stand in for both.
Rebuild from the tag, or trust the build platform?
Rebuild when the project's builds are reproducible; otherwise verify the provenance, and have the benchmark say which kind of verification each task supports. The Reproducible Builds project defines a build as reproducible when, given the same source code, build environment and build instructions, anyone can recreate bit-by-bit identical copies of the specified artifacts. When a project meets that bar, a reviewer can rebuild from the tag and compare digests without trusting the original build system.
Many projects do not, so the fallback is verifying the provenance. SLSA's guidance is to check the envelope signature, check that the statement's subject matches the artifact digest, and compare the provenance against expected values for at least the builder identity, the canonical source repository, the build type and the external parameters. That trusts the build platform rather than the rebuild, which is a weaker claim and should be labeled as such in the task.
Record the verification mode per task: rebuilt and matched, provenance verified, or neither. A benchmark that mixes them without labels will score agents for confidence they could not have earned from the evidence given.
What has to come out before the release history leaves the company
A release benchmark built from a real product needs the same screening as any code dataset, plus a few items specific to releases. Proprietary source is the obvious one: the company must own it or have the right to license it, and third-party dependencies keep their own licenses, which do not transfer with your repository.
Build logs and CI configuration often contain tokens, internal hostnames and signing details that must be removed. Release notes and pull requests can name customers who requested fixes. Security entries can describe vulnerabilities that are fixed in the current release but still exploitable in versions customers run, so embargoed or recent security changes should be held back or reduced to their verdict label.
Contributor names and emails in commit metadata are personal data in many jurisdictions; pseudonymize them consistently so the authorship structure survives. Keep the redaction record with each task so a reviewer can see what was changed and confirm that verdicts did not depend on removed content.
If your release records are complete enough to license
Software companies that kept tags, build provenance, deployment logs and flag history alongside their release notes hold records that are hard to assemble any other way. A software company that decides to license this history can do it through SourceX, which runs rights review, contracting, preparation and delivery around the dataset.
Buyers usually look for several years of consistent history in one system and a substantial volume of records, although unusual work can qualify with less, and each case is assessed individually. The company signs off on scope and terms before anything is delivered, and a short assessment, with no repository or log upload, tells it whether it qualifies.
Frequently asked questions
What is release note provenance?
Release note provenance is the evidence that ties each statement in a release note to the change it describes, the commit that was tagged, the build that used that commit and the artifact that was deployed. Without it, a release note is an unverified summary.
Are automatically generated release notes accurate?
They accurately list merged pull requests in the tag range, but their categories depend on labels, and configured exclusions can drop changes. GitHub's own documentation tells users to check generated notes before publishing, so treat them as a starting list, not proof.
How do you check that a deployed artifact matches its source?
Compare the deployed artifact's digest with the subject digest in its build provenance, then check that the provenance records the tagged commit as its source. If the build is reproducible, rebuilding from the tag and comparing digests is a stronger check.
What should happen when a released version is rebuilt?
Semantic Versioning says a released version's contents must not be modified and changes must ship as a new version. A second artifact with a different digest under the same version should be flagged and treated as unverifiable until provenance and deployment records explain it.
Can release history be used to train or evaluate AI agents?
Yes, when the company owns or controls the code and records and screens out secrets, customer names, personal data and sensitive security details. The useful unit is a release with its notes, commit range, provenance and deployment record linked together.
Sources
- SLSA build provenance records external parameters and resolved dependencies, such as the specific git commit a repository reference resolved to, and identifies build outputs as the subject. Source
- SLSA verification checks the signature, that the statement subject matches the artifact digest, and compares builder identity, source repository, build type and external parameters with expected values. Source
- The in-toto Statement requires every subject to have a digest and treats subjects as immutable. Source
- GitHub's automatically generated release notes list merged pull requests, contributors and a full changelog link; release.yml can categorize by label and exclude labels or authors; users should check generated notes. Source
- GitHub releases are based on Git tags, and a tag date may differ from the release date. Source
- Keep a Changelog says changelogs are for humans and groups changes as Added, Changed, Deprecated, Removed, Fixed and Security, with an Unreleased section. Source
- Semantic Versioning defines patch, minor and major increments and says a released version's contents must not be modified. Source
- Conventional Commits correlates fix with a patch, feat with a minor version and BREAKING CHANGE with a major version. Source
- The Reproducible Builds project defines a build as reproducible when the same source, environment and instructions let anyone recreate bit-by-bit identical artifacts. Source
See if your company qualifies
A short company assessment. No data uploads are needed.