AI uses for records
Software dependency upgrade dataset: real upgrades as coding tasks
By SourceX Editorial · Updated
9 sections
Short answer
A software dependency upgrade dataset turns finished upgrades into multi-constraint coding tasks: freeze the repository before the change, list every constraint in force (version ranges, lockfile, runtime, resolver conflicts, advisories, breaking APIs), keep the human patch as the reference answer, and prove it with tests that fail before and pass after. Add review rationale and a dated outcome so each task can be rebuilt and verified.
Record timeline
Software dependency upgrade
Key takeaways
- The valuable upgrades are the ones where several constraints collided and a person had to change code.
- Every constraint in a task should cite a file or record in the starting state.
- Lockfiles, runtime versions and pinned CI images are what make a task rebuildable months later.
- Fail-to-pass tests must flip reliably on rerun; flaky flips are excluded, not counted.
- A merged status is not an outcome; deploys, reverts and incidents dated after merge are.
How can dependency upgrades become realistic multi-constraint coding tasks?
Take an upgrade your team already finished, freeze the repository at the commit before it, and record every constraint the engineer had to satisfy at once: the manifest ranges, the lockfile, the language runtime, transitive conflicts, any security advisory and the breaking API changes. The human patch is the reference answer, and the tests that failed before it and passed after it are the check.
That combination is what makes the task realistic. A single version bump is trivial; an upgrade where a security fix sits in a new major release, a sibling library caps the same dependency below that release, and the runtime has to move first is the kind of problem engineers spend days on. The record of how your team untangled it is the evidence a software dependency upgrade dataset needs.
The public precedent is SWE-bench, whose authors built 2,294 software engineering problems from real GitHub issues and the pull requests that resolved them across 12 popular Python repositories, using the repository's tests to judge a model's edit. Upgrade tasks follow the same pattern with a different trigger: the starting point is a version change rather than a bug report.
Where do the constraints in an upgrade come from?
Every constraint in an upgrade task should point to a file or record in the starting state, not to someone's memory. The table lists the usual sources and what each one forces the solver to do.
Two of these deserve a note. Semantic Versioning requires the major version to increase for any backward-incompatible change to the public API, which is why advisory fixes that land only in a new major release turn a security patch into a code change. And Renovate's documentation states that by default it does not apply language constraints to upgrades, so a bot-proposed version may need a runtime the project does not run yet.
| Constraint | Where it is recorded | What it forces | Evidence to keep |
|---|---|---|---|
| Declared range | Manifest: package.json, pyproject.toml, requirements file, go.mod | The new version must satisfy or deliberately change the range | Manifest at the parent commit and after the patch |
| Resolved versions | Lockfile: package-lock.json, poetry.lock, Gemfile.lock | Sub-dependencies move only when forced or regenerated | Lockfile diff, not just the manifest diff |
| Runtime or language version | engines field, python_requires, toolchain files, CI images | The runtime may have to change before the library can | CI image tag or digest and runtime version used in each run |
| Transitive conflict | Resolver output from npm, pip, Bundler or Cargo | A sibling package may cap the dependency below the needed release | The resolver error text, such as pip's ResolutionImpossible message |
| Security advisory | Advisory record, such as an OSV entry with introduced and fixed events | The target version must be at or beyond the fixed event | Advisory ID and the affected range at the time |
| Breaking API change | Release notes, changelog, migration guide | Call sites, configuration or types must change | The release notes version consulted and the code diff |
| Team update policy | dependabot.yml, renovate.json, contributing rules | Grouping, pinning or a versioning strategy limits the acceptable patch | Policy file at the parent commit |
Which upgrades in your history are worth turning into tasks?
Keep upgrades that needed human judgment and drop the ones a bot finished alone. A routine bump that merged green on the first run has a correct answer, but it teaches little, because the only edit was a version string.
Dependabot's own options show why the distinction is visible in the record. Its versioning strategies include lockfile-only, which only updates lockfiles and ignores versions that would need manifest changes, alongside increase and widen, which edit the manifest's requirements. A pull request that stayed lockfile-only and passed is a weak task; one where a person had to change code after the bot's first attempt failed is a strong one.
| Upgrade pattern | Signal in the history | Task value | Decision |
|---|---|---|---|
| Bot bump, lockfile only, green first run | One commit by the bot, no human code changes | Low: the answer is a version string | Keep a sample as negative or easy controls only |
| Range widened or raised, green first run | Manifest and lockfile diff, no code diff | Low to medium | Keep when the range change itself was debated in review |
| Major upgrade with code changes | Failing CI on the bump, then human commits touching call sites | High | Keep as a core task |
| Advisory-driven upgrade across a major version | Advisory ID in the PR, conflict with a sibling package | High: several constraints at once | Keep and record the advisory range |
| Runtime upgrade cascade | Runtime change PR followed by dependent library upgrades | High but long | Split into linked tasks with shared starting state |
| Upgrade later reverted | Revert commit or rollback record after deploy | High as a counterexample | Keep with the revert as the outcome |
| Abandoned upgrade PR | Closed without merge, discussion of blockers | Medium: shows why a path failed | Keep only when the blocker is recorded in the thread |
What should the version constraint patch and regression evidence packet include?
The packet is one folder per task with a fixed manifest, so a reviewer can rebuild the starting state, read the constraints, apply the reference patch and rerun the checks without asking anyone. Use the fields below as the manifest; every entry points to a file or record ID.
Keep the manifest diff, the lockfile diff and the code diff as three separate files. A solver that only edits version strings should fail the task, and separating the diffs makes it obvious which part of the reference answer carried the engineering work. Store test commands exactly as CI ran them, including environment variables that change behavior, with secret values removed.
| Packet field | Contents | Source in your systems | Verification check |
|---|---|---|---|
| Task ID and trigger | Why the upgrade started: advisory, end of support, feature need, bot proposal | Ticket, PR description, advisory ID | Trigger is a record, not a summary written later |
| Starting commit | Repository state at the parent of the first upgrade commit | Git history | Checkout builds and the pre-patch test result reproduces |
| Constraint list | Each constraint from the table above with its file and line | Manifests, lockfiles, CI config, policy files | Every constraint cites a file at the starting commit |
| Environment | Runtime version, OS image digest, package registry snapshot or mirror | CI configuration and build logs | A second machine resolves the same versions |
| Reference patch | Manifest, lockfile and code diffs, kept separate | Merged PR commits | Patch applies cleanly to the starting commit |
| Fail-to-pass tests | Tests that fail before the patch and pass after | CI runs on the bump and on the final commit | Rerun shows the same flip at least twice |
| Pass-to-pass tests | Tests that passed before and must still pass | Full CI suite | No previously passing test fails after the patch |
| Review rationale | Review comments explaining choices and rejected options | PR review threads | Rationale names the constraint it addresses |
| Outcome evidence | Deploy record, revert or incident within a set window | Deploy logs, incident tracker | Outcome is a record dated after merge, not the PR status |
| Rights and exclusions | Code ownership note, removed secrets, pseudonymized authors | Legal and security review | Signed off before the packet leaves the repository |
Worked case: a security fix that only exists in the next major release
This case is illustrative; the package names and counts are invented to show the shape of one task, not taken from a real repository. A service pins an HTTP client library at version 2.x through a caret range. An advisory lists the vulnerable range as introduced at 2.0.0 and fixed at 3.1.0, so no 2.x release closes it.
Under node-semver's definition, a caret range allows changes that do not modify the left-most non-zero element, so ^2.4.0 can never resolve to 3.1.0. The upgrade therefore needs a manifest change, and the move to 3.x brings the breaking changes that Semantic Versioning signals with a major increment. A tracing library in the same service declares a peer range capped below 3.0.0, which is the second constraint, and its own fix requires a newer runtime than the CI image runs, which is the third.
Read the table from top to bottom as the order a careful engineer would work in. The point of keeping the closed lockfile-only attempt is that it is the answer a weak solver will produce first: it looks like progress, it may even pass a narrow test run, and it leaves the vulnerable version installed. A task that cannot tell those two answers apart is not testing upgrade skill.
| Stage | Evidence in the packet | What the solver must do | How it is checked |
|---|---|---|---|
| Starting state | Commit before the bot PR; manifest with ^2.4.0; CI image on the older runtime | Reproduce the build and the vulnerable resolution | Lockfile resolves 2.x; advisory scanner flags it |
| Constraint 1 | Advisory record: introduced 2.0.0, fixed 3.1.0 | Choose a target at or above 3.1.0 | Resolved version is outside the affected range |
| Constraint 2 | Resolver error showing the tracing library's peer cap below 3.0.0 | Upgrade or replace the tracing library too | Resolver completes without a conflict |
| Constraint 3 | Tracing library release notes requiring the newer runtime | Move the CI image and runtime first | Build runs on the new image digest |
| Code change | Release notes for 3.x listing renamed options and a changed retry default | Update call sites and set retries explicitly | Illustrative count: 3 fail-to-pass tests flip; all pass-to-pass tests hold |
| Decision record | Review thread rejecting a fork of the 2.x line | None; this is context for evaluation | Rationale names constraint 1 |
| Outcome | Deploy record and 14 days with no related incident or revert | None | Outcome is dated after merge |
| Counterexample | An earlier PR that only bumped the lockfile and was closed | Shows a tempting wrong path | Kept as a failed attempt with its CI log |
Can someone outside the team rebuild an upgrade task months later?
Only if the packet records the resolved versions and the environment, not just the intent. Renovate's documentation explains that a lock file kept in sync with package.json locks the exact dependencies and sub-dependencies, but that once it falls out of sync the manifest becomes the source of truth for installs. A packet without its lockfile will resolve different versions on the day someone rebuilds it.
Rebuild every packet on a clean machine before it counts. Pin the CI image by digest, record the runtime version, and keep a mirror or snapshot of the packages that were resolved, because registries remove or yank versions over time. Rerun the fail-to-pass tests more than once; a test that flips only some of the time is a flaky test, not evidence, and should move to an excluded list with its rerun record.
Resolver errors are worth keeping verbatim. pip's documentation shows the shape of a ResolutionImpossible error, naming each package and the conflicting requirement it places on a shared dependency. That text is a precise statement of a constraint and a far better task input than a paraphrase.
What proves an upgrade worked after the pull request merged?
The merged pull request proves only that the code was accepted. The outcome is proved by records dated after the merge: the deploy that shipped it, the absence or presence of a revert, and any incident traced back to the change within a window you define in advance.
Write the window into the packet before you look at the results, for example 14 days after deploy, and apply it to every task. A task whose outcome is a merged status and nothing else should be labelled as unverified rather than successful. Reverted upgrades are not failures for a dataset; with the revert and its reason attached, they are some of the most instructive tasks you have.
- Decision evidence: review comments that name a constraint, rejected alternatives, and who approved.
- Outcome evidence: deploy ID and time, revert commit if any, incident IDs linked to the change.
- Not evidence on its own: PR status, a green badge, a closing comment such as works now.
What has to be cleared before upgrade history leaves the company?
Clear three things: who owns the code, what secrets the history contains, and what personal data rides along with commits and reviews. Your own application code and review threads are usually yours to license, but vendored third-party code and copied snippets carry their own licenses, so an open-source audit belongs before packaging.
Upgrade work leaks secrets in predictable places: private registry URLs with tokens in lockfiles or config, credentials printed in CI logs, and environment files committed during a rushed fix. Scan the full packet, including logs, and rotate anything found rather than only deleting it from the copy. Commit author names and emails, and reviewer handles, are personal data in most privacy regimes; replace them with stable pseudonyms that keep who-reviewed-whom structure intact.
Bringing upgrade packets into a licensing review
A few hundred well-built upgrade packets with real constraints and outcomes say more about engineering judgment than thousands of bot bumps. Being able to export them, though, is not the same as having the right to license them; customer contracts, contributor agreements and third-party code each need their own review, and pseudonymization does not settle ownership.
Upgrade packets that pass this bar can go into a licensing review, where SourceX does the qualification, rights review and contracting and then prepares, delivers and pays for the licensed dataset. A short company assessment comes first, with no code or data uploaded, and nothing is delivered until the company has approved the scope and terms and signed.
Frequently asked questions
What is a software dependency upgrade dataset?
It is a set of tasks built from real upgrades, each with the repository state before the change, the version constraints in force, the patch engineers wrote and tests that show the result. Good sets also carry the review rationale and a dated outcome such as a deploy or a revert. The point is that every claimed success can be rerun and checked.
Are Dependabot or Renovate pull requests useful as training tasks?
Some are. Bumps that changed only a lockfile and passed on the first run are easy and mostly useful as controls. Bot pull requests that failed CI and then needed human code changes, especially across a major version, are the valuable ones because the human follow-up commits are the answer.
Why does the lockfile matter if the manifest shows the new version?
The manifest usually holds a range, while the lockfile records the exact versions of every direct and indirect dependency that were installed. Without it, a rebuild months later resolves different versions and the before and after test results no longer reproduce. Keep both, and keep their diffs separate from the code diff.
How is this different from SWE-bench style issue tasks?
SWE-bench tasks start from an issue report and a pull request that fixed it. Upgrade tasks start from a version change and add constraints that issue tasks rarely carry, such as runtime requirements, resolver conflicts and advisory ranges. The verification idea, tests that fail before the patch and pass after it, is the same.
What makes an upgrade task verifiable rather than plausible?
A pinned environment, a reference patch that applies cleanly, fail-to-pass tests that flip reliably on rerun, pass-to-pass tests that still pass, and an outcome record dated after the merge. If any of those is missing, label the task as unverified instead of counting it as solved.
Sources
- SWE-bench is a benchmark of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories, where a model's patch must pass tests related to the issue. Source
- Semantic Versioning 2.0.0 requires the major version to be incremented when backward-incompatible changes are introduced to the public API. Source
- In node-semver, a caret range allows changes that do not modify the left-most non-zero element of the major, minor, patch tuple. Source
- Dependabot's versioning-strategy option includes increase, increase-if-necessary, lockfile-only (only update lockfiles, ignore versions needing manifest changes) and widen. Source
- Renovate by default does not apply language constraints to upgrades and will propose any stable upgrade without checking the project's language version. Source
- A lock file kept in sync with package.json locks exact dependencies and sub-dependencies; out of sync, package.json becomes the source of truth for installs. Source
- pip reports a ResolutionImpossible error when requested packages have conflicting dependencies and names the conflicting requirements. Source
- The OSV schema describes affected version ranges as a timeline of events, including introduced and fixed versions. Source
See if your company qualifies
A short company assessment. No data uploads are needed.