AI uses for records
Code issue alignment dataset: what 36 re-fixed Node.js issues show
By SourceX Editorial · Updated
10 sections
Short answer
A code issue alignment dataset links what an issue asked for to the change that claimed to fix it, that change's tests and review, and what happened after merge. In Node.js, 36 of 1,580 issues first fixed in 2020-2024 under rules requiring a passing test and CI were claimed again by a later pull request. Of the 31 later fixes that were new changes, 8 show the first change missed the actual problem.
Agent run
Code issue alignment dataset
Key takeaways
- Passing tests are one field in an alignment record, not its label; the label needs a later record and a reader.
- In Node.js, 2.3 percent of issues first fixed in 2020-2024 were claimed again by a different pull request, a floor on fixes that did not hold.
- Of the 31 later fixes that were new changes, 12 covered a case the first missed, 7 went after another cause, 4 replaced a reverted fix and 1 followed a changed specification.
- A re-fix flag alone is noisy: one Node.js issue was claimed twice only because two pull requests kept the contributing guide's example link.
- Buyers should ask for post-merge events, the fixer's diagnosis, the follow-up rationale and the search window with every record.
What is a code issue alignment dataset?
A code issue alignment dataset pairs what an issue asked for with the change that claimed to resolve it, the tests and review around that change, and what happened after the merge, then labels whether the change addressed the stated problem. In such a record, a passing test suite is one field. The verdict comes from later evidence and from someone reading it.
The decision it supports is practical. An AI team buying training or evaluation data for coding models has to know whether a supplier's issue and patch records can tell a model, or a grader, the difference between a change that passes and a change that resolves. Issue-to-fix pairs usually treat the merged patch as the gold answer, and review preference data ranks competing patches. Neither asks whether the gold answer held up.
Settling that takes a record that contradicts the first fix: a revert, a reopened issue, a second change that claims the same issue, or a specification the project later changed, plus a reader who explains what the contradiction means. This article counts such records in one large public project, labels each one, and turns the result into a trace format and an outcome rubric a buyer can put in a data request.
Why a green test run is a weak label for solved
Because tests check what their authors thought to check, and research on automated repair has shown for a decade how far a passing patch can drift from a correct one. In their 2015 ISSTA paper, Qi, Long, Achour and Rinard found that most patches accepted by three generate-and-validate repair systems were equivalent to deleting functionality, and that GenProg produced a correct patch for only 2 of the 105 defects considered.
The gap persists with language-model agents. Wang, Pradel and Liu (arXiv 2503.15223, revised September 2025) report that SWE-bench's validation counted 7.8 percent of patches from three issue-solving tools as correct even though they failed the developers' own test suite, and that 29.6 percent of plausible patches behaved differently from the ground-truth patch under differential testing; manual inspection found 28.6 percent of those divergent patches certainly incorrect.
Human judgment widens the gap further. In a note dated March 10, 2026, METR reported that four active maintainers from three SWE-bench Verified repositories reviewed 296 agent pull requests that had passed the automated grader and would not have merged about half of them; average maintainer merge decisions ran about 24 percentage points below the grader's scores. The same maintainers merged about 68 percent of the original human-written patches, so the reference answers do not all survive a second look either.
Each of these studies grades a patch against a reference assumed to be right. For a buyer, the harder question is how often the reference itself was wrong, and only a project's own history can answer it.
How often does a tested Node.js fix get fixed again?
Rarely, but often enough to measure. Of 1,580 Node.js issues whose first fix landed between 2020 and 2024, 36, or 2.3 percent, were later claimed by a commit from a different pull request, and 31 of those later claims were new changes rather than the same patch landed again.
Node.js suits the test because its rules make a fix claim strong. Its collaborator guide says a pull request must pass code review and CI before landing, and that all fixes must have a test case which demonstrates the defect, failing before the change and passing after it. Its pull request guide tells contributors to use the Fixes: prefix with the full issue URL when a patch fixes an open issue and Refs: for other references. A Fixes: line is therefore the project's own statement that a reviewed, tested change resolved an issue.
The count reads every commit message on Node.js's main branch at commit 75191c4 (October 27, 2026). The median time from the first fix to the second new change was 35.1 days; 13 of the 31 arrived within 30 days and 6 more than a year later. The figure is a floor: a fix that missed often comes back as a fresh issue rather than a second claim on the old one, and those cases are invisible to this rule. The steps and the rule behind each count:
| Step | Count | Rule applied |
|---|---|---|
| Commits on the main branch at 75191c4 | 49,259 | Every commit reachable from main on 2026-10-27 |
| Issues named on a Fixes: line | 4,473 | nodejs/node issue URLs only; 590 other targets dropped and listed |
| Issues first claimed in 2020-2024 | 1,580 | Earliest claim's committer date in the window, UTC |
| Claimed again by another pull request | 36 | A later claim with a different PR-URL, searched to 2026-10-27 |
| The same change landed again | 5 | Same subject line, or a reland marker in the later commit |
| A new change claimed the fix | 31 | Every one read and labeled in the next section |
Seven labels for what a second fix reveals
Reading both commit messages for all 31 new changes gives seven outcomes, and only two of them mean the first change missed the problem it claimed to solve. Seven later changes went after a different cause than the first, and one followed a decision to change the specification instead of the code: 8 of 31 in all.
The largest group is narrower. Twelve later changes kept the first diagnosis and covered a sibling function, option, platform or path the first change left out. Four replaced a first change that had been reverted, and four give no reason a reader can label. Two were planned parts of a multi-step issue, and one pair of claims was a linking mistake. The rubric, with counts and one Node.js example per label:
| Label | What the later record shows | Of 31 | Node.js example | Use for a buyer |
|---|---|---|---|---|
| Case missed | Same cause; the first change left out a sibling API, option, platform or path | 12 | Issue 44720: the async copy filter was fixed first, its sync twin 21 days later | Training data for checking sibling code paths before claiming a fix |
| Other cause | The failure persisted because the first change targeted a different mechanism | 7 | Issue 44143: the flaky test was rewritten first, the library bug fixed a day later | Negative examples, and graders that ask for the diagnosis, not just a green run |
| Withdrawn and redone | The first change was reverted and a different change later claimed the fix | 4 | Issue 42713: an HTTP/2 fix was reverted, then redone after a fresh cause analysis | Pairs of a failed approach and its replacement, with the revert reason |
| Not stated | The later commit gives no reason that supports a label | 4 | Issue 25988: a second test deflake with no message body | Hold back until the issue thread is read |
| Staged delivery | A multi-part issue; the first change delivered one planned part | 2 | Issue 50571: benchmark iterations fixed file by file across 15 pull requests | Keep out of wrong-fix labels; useful as task decomposition records |
| Requirement reread | The project changed what correct behavior means | 1 | Issue 32074: a certificate fix was reverted and the specification narrowed | Evaluation items whose reference answer itself changed |
| Mislinked | The claim points at an unrelated issue | 1 | Issue 1337: two unrelated pull requests copied the guide's example link | Linkage checks before any label is trusted |
Three records where the passing change addressed something else
Three of the eight show how a change can satisfy every gate and still answer a different question. Each passed review and CI under the rules above, and each was overtaken by a record that named the real problem.
A test that hid a library bug. In August 2022, a pull request claiming issue 44143 changed a flaky diagnostics-channel test to stop using deprecated APIs; its author wrote that the failure could not be reproduced locally. The next day a separate change claiming the same issue fixed the diagnostics channel library itself. The first change made a test quieter; the second fixed the code the test was meant to protect. The label a model should learn is Other cause, and the teaching point is that the test file was never the defect.
A fix the project decided it did not want. In March 2020, a change claiming issue 32074 made tls.rootCertificates include certificates loaded from NODE_EXTRA_CA_CERTS, as the issue asked. In May the change was reverted; the revert message says it caused non-deterministic behavior with OpenSSL system or file-based certificate stores, and that the safest action was to "change the specification for tls.rootCertificates to state that it only returns the bundled certificates instead of the current ones." A documentation change then claimed the issue. The first patch was correct against the requirement as written; the requirement moved.
A workaround documented in place of a bug. In May 2024, a change claiming issue 53176 added a documentation note about passing a job-count flag to the Ninja build on macOS. On June 8, another change claiming the same issue fixed missing spacing before NINJA_ARGS in the build itself. For about a week users had a workaround while the build bug sat underneath it. For training data, the pair teaches the difference between telling people how to avoid a failure and removing it.
The engineering decision trace: fields a usable record needs
A record is usable when a second reader can see what was asked, what the fixer believed, what the tests checked and what happened afterward, all tied to dates. Most of the fields below already exist in issue trackers, code review tools and CI systems; the work is linking them and fixing the snapshot time.
Two fields decide whether the label can be trusted. The issue statement must be captured as it stood when the fix merged, because issues are edited after the fact. The post-merge search must state its window and its rule, because a record with no later events means either that the fix held or that nobody looked. The decision trace, field by field:
| Field | Where it comes from | Why the label needs it | Acceptance check |
|---|---|---|---|
| Issue statement at merge time | Issue body and comments up to the merge timestamp | The requirement the change is judged against | Snapshot taken no later than the merge |
| Reproduction and expected behavior | Issue text or an attached failing case | Separates the reported failure from the fixer's reading of it | Present, or marked missing |
| Fixer's diagnosis | Pull request description and commit body | Shows which cause the change targeted | Kept verbatim, not summarized |
| Change and its tests | Diff, plus tests added or edited | Shows what passing actually checked | Test fails on the parent commit and passes on the change |
| Review and CI outcome | Review votes, comments and final CI status | The gate the change cleared and objections raised before merge | Comments kept with reviewer names removed |
| Link type and grouping key | Fixes or Refs line and pull request number | Separates a claimed fix from a partial step and groups multi-commit landings | Recorded per issue and per pull request |
| Post-merge events | Reverts, reopen events, later claims, new issues citing the old one | The evidence that the fix did or did not hold | Window length and search rule stated |
| Label and evidence pointer | A reader's rubric label and the record behind it | Makes the verdict auditable and open to relabeling | Label from the published set, with labeler role and confidence |
How buyers would use alignment records for training and evaluation
As contrast pairs for training and as held-out traps for evaluation. The first and second fix for the same issue, with the label and the follow-up rationale, form a preference pair that carries a reason, which is rarer than a ranking. Because Case missed is the largest group, these pairs mostly teach a model to look for the sibling function, option or platform before declaring a fix complete.
For evaluation, the re-fixed issue is the useful item. Give an agent the issue as it stood before the first fix, then run its patch against two test sets: the tests added with the first fix and the tests added with the second. A patch that passes only the first set has reproduced the original miss. Differential testing in the style of PatchDiff, comparing an agent's patch with the final accepted fix on generated inputs, catches behavior differences neither test set covers, and a maintainer-style review on a sample, as in METR's study, checks what tests cannot.
Public histories carry a known limit here. Popular open-source repositories are likely in pretraining corpora, so a held-out evaluation built from Node.js may partly measure recall; private repositories avoid that, at the cost of rights review and de-identification. Records that pair a code change with its review rationale overlap with code review preference data, and buyers can combine the two.
The strongest objection: most second fixes are ordinary iteration
Yes, and the count supports it: 19 of the 31 later changes are not evidence of a wrong fix. Twelve cover a missed case, four give no reason, two are planned stages and one is a linking error. A team that labeled every re-fixed issue as a wrong-problem example would be wrong more often than right.
The linking error is instructive. Issue 1337 was claimed by a 2020 REPL change and a 2023 HTTP/2 change that have nothing to do with each other; the number matches the example link in Node.js's own pull request guide, which suggests both authors left the example in place. Any pipeline that trusts Fixes: lines without a check will import errors like this one.
Two more limits cut the other way. The revert flag in the count is a lower bound: the revert behind issue 32074 lacks the usual "This reverts commit" line and was caught only because its subject quoted the original's. And the labels come from commit messages alone, read by one person; a reader with the issue threads might move several from Case missed to Other cause or back. The practical conclusion holds either way: buy the raw events and the rationale text, and budget for human labeling rather than treating a re-fix flag as the label.
What private engineering histories add, and how to scope a request
The requirement and the outcome, which public projects rarely record. In a company, an issue often starts as a customer ticket or a product decision, and a fix that missed shows up as a repeat incident, a support escalation or a rollback, records that sit in systems outside the repository. Linking those to the code change gives the label a business meaning that a Fixes: line cannot carry.
Those records also carry more risk: customer data in tickets, secrets in logs, employee names in reviews and contract limits on what the company may license. Rights review and de-identification come before any record leaves the company.
SourceX sources datasets from businesses on request rather than from stock, and buyers describe scope, volume, language, format, timing and licensing needs through its data request form. For alignment records, a useful request also names the intended use, the post-merge window, the required trace fields and an acceptance test, such as a minimum share of records with a captured issue snapshot and a stated search rule. Whether a given company's history fits is decided case by case.
Methods note: counting rules, labels and limits
Counts come from a script that reads commit messages from a bare clone of the nodejs/node main branch (commits only, no file trees) at commit 75191c45afeb, committed October 27, 2026, through git log; no repository code was run. A fix claim is a commit-message line beginning "Fixes:", matched case-sensitively at the very start of the line, that contains a URL of the form https://github.com/nodejs/node/issues/N, optionally with www, where N is followed by the end of the line, whitespace or one of # ? . , : ; ). The script prints every other Fixes: target it drops, 590 in all, mostly issues in other repositories, pull request links, bare #N references and HackerOne reports. Commits whose subject starts with 'Revert "' make no claims; there are 401.
Claims are grouped by the last PR-URL line in each commit, because Node.js copies a description's Fixes: lines onto every commit of a multi-commit pull request. The cohort uses committer dates, not author dates: it is every issue whose earliest claim has a UTC committer date from 2020-01-01 through 2024-12-31; a re-fixed issue has a later claim from a different pull request. The later change counts as the same change when its subject matches the first, or its subject or body carries a reland marker; otherwise it is a new change. A first fix counts as reverted when a revert names its hash or pull request on a "This reverts" line, or quotes its subject exactly. Gaps are exact committer-timestamp differences in days; medians and the share are rounded half up to one decimal, once.
All 31 new changes were labeled by reading both commit messages, not issue threads, with this precedence when several labels fit: Mislinked, Requirement reread, Withdrawn and redone, Other cause, Case missed, Staged delivery, Not stated. The saved files hold issue numbers, commit hashes, dates, flags and labels, with no names. Not measured: issue threads, reopen events, issues filed anew after a miss, test contents, CI logs, and any agent patch. The external figures are the cited authors' own and were verified through search excerpts on their publishers' sites.
Frequently asked questions
What is the difference between issue-to-fix pairs and a code issue alignment dataset?
Issue-to-fix pairs treat the merged patch as the correct answer to the issue. A code issue alignment dataset adds what happened after the merge and a label saying whether that patch addressed the stated problem. The extra fields matter most for evaluation, where a wrong reference answer quietly penalizes good patches.
Can passing tests prove a code fix solved the issue?
No. Tests confirm the behavior their authors chose to check. In Node.js, which requires a failing-then-passing test and passing CI for every fix, 36 of 1,580 issues first fixed in 2020-2024 were later claimed by another pull request, and 8 of the 31 later changes that were new show the first change missed the problem it claimed.
How common are code fixes that solve the wrong problem?
Uncommon in a well-run project, and hard to see without reading. The Node.js count gives a floor: 8 wrong-problem cases among 1,580 issues in a five-year cohort, found through second fix claims alone. Misses that came back as new issues are not counted, so the true rate is higher by an unknown amount.
What labels should a code issue alignment dataset use?
Use a small published set and make labelers cite the record behind each choice. This article's rubric has seven: Case missed, Other cause, Withdrawn and redone, Staged delivery, Not stated, Requirement reread and Mislinked. Only Other cause and Requirement reread mean the first change answered a different question than the issue asked.
Can re-fixed issues be used to evaluate coding agents?
Yes, as held-out items. Give the agent the issue as it stood before the first fix and run its patch against the tests added with both the first and second fixes; passing only the first set reproduces the original miss. Prefer private repositories where possible, since public ones may sit in pretraining data.
Sources
- Node.js commit history on main at commit 75191c4 (October 27, 2026), from which SourceX counted 1,580 issues first claimed by a Fixes: line in 2020-2024, the 36 claimed again by another pull request and the labels of the 31 new changes. Source
- Node.js collaborator guide: a pull request must pass code review and CI before landing, and every fix needs a test that fails before the change and passes after it. Source
- Node.js pull request guide: use the Fixes: prefix and the full issue URL when a patch fixes an open issue, Refs: for other references; trailers are copied to each commit of a multi-commit pull request; the example link uses issue 1337. Source
- METR, March 10, 2026: maintainers judged about half of 296 grader-passing agent pull requests unmergeable, about 24 percentage points below the grader, and merged about 68 percent of the original human patches. Source
- Wang, Pradel and Liu: 7.8 percent of patches counted correct while failing developer tests; 29.6 percent of plausible patches behave differently from ground truth; 28.6 percent of those are certainly incorrect. Source
- Qi, Long, Achour and Rinard, ISSTA 2015: most accepted repair patches were equivalent to deleting functionality, and GenProg produced a correct patch for 2 of 105 defects. Source
- SourceX's data request page: datasets are sourced on request, not held in stock, and buyers share scope, volume, language, format, timing and licensing. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.