AI uses for records
Issue history code patch experiment: a design and a 237-task pilot
By SourceX Editorial · Updated
10 sections
Short answer
An issue history code patch experiment holds the candidate patches fixed and changes one thing: whether the selector also reads the issue's pre-fix comments. In a public pilot on 237 SWE-bench Verified tasks, adding those comments to a lexical selector moved the chosen patch's pass rate from 59.81 to 60.06 percent, a 0.25-point difference with an interval from -2.2 to 2.8 points. Model-based selectors were not run.
Record timeline
Issue history code patch
Key takeaways
- Hold the candidate pool fixed and vary only what the selector reads: issue text alone, or issue text plus comments posted before the fix began.
- Public leaderboards cannot answer the question, because SWE-bench asks entrants not to use the hints field that holds those comments.
- In a frozen, single-run pilot on 237 tasks, comments moved a lexical selector's pass rate by 0.25 points, with an interval from -2.2 to 2.8.
- The visible gain sat in the 43 tasks whose comments quote the fix, so leak checks have to be reported apart from judgment.
- A full run on licensed history should label candidates by what happened after merge, not only by tests.
What does an issue history code patch experiment test?
An issue history code patch experiment tests one claim: that a system choosing among candidate patches for the same issue picks a working patch more often when it can read the issue's discussion history, not just the issue text. Everything else is held fixed, including the candidates, the selector and the grading, so any difference comes from the history alone.
Patch selection is a real stage in coding agents. The Agentless project's README describes generating several candidate patches, running regression tests and a generated reproduction test, and using the results to re-rank the remaining patches and pick one to submit. SWE-bench's own submission checklist allows a Best@k system, in which a distinct module of the system decides which attempt to submit without using the benchmark's evaluation or its fail-to-pass and pass-to-pass tests.
For a buyer the decision is practical: whether to pay for issue threads, review discussion and ticket comments alongside code, or whether issue titles and descriptions are enough. This page publishes the design for answering that, runs the one arm that can be run reproducibly on public data, and states plainly what has not been run.
Why public leaderboards leave the question open
Public leaderboards cannot settle it because their rules keep issue history out of the systems they rank. SWE-bench's submission checklist asks entrants to confirm that the system does not use the hints field, and its note on test use repeats that a system must not use the pass-to-pass tests, the fail-to-pass tests or the hints_text field.
The history exists in the benchmark, though. SWE-bench's documentation describes each task directory as holding problem_statement.md, the issue text shown to a model, and hints.md, discussion from the issue, absent when there is none. The collection code keeps only comments from before the fix began: it takes the time of the first commit in the fixing pull request and keeps GitHub issue comments whose last update came earlier, or, for Django's Trac tickets, ticket comments timestamped earlier.
That makes a clean experiment possible. Every candidate on the leaderboard was generated without the comments, so the comments can be added at selection time without having shaped the candidates. In the public task data, 335 of the 500 SWE-bench Verified tasks carry non-empty comments.
The experiment design: hypothesis, held-out set, comparison, metric and stop rules
The design fixes seven elements before any result is seen. The middle column is what the public pilot fixed and applied; the right column is what a full run on licensed company data should use, where the candidates, the history and the outcome labels are richer than a public benchmark's.
| Element | Fixed for the public pilot | For a full run on licensed history |
|---|---|---|
| Hypothesis | With the candidate pool held fixed, a selector reading issue text plus pre-fix comments picks a test-passing patch more often than the same selector reading issue text alone | The same, stated for each selector before any run |
| Unit and pool | A SWE-bench Verified task with comments and at least one passing and one failing candidate; every non-empty patch from 73 leaderboard submissions | An issue with two or more candidate fixes: review revisions, abandoned pull requests, agent outputs |
| Comparison | Arm A reads problem_statement.md; arm B reads it plus hints.md; same scorer, same pool | Same two arms; candidate order shuffled with a published seed for model selectors |
| Held-out set | Tasks created on or after January 1, 2022 (75 in the primary set), reported separately | Issues filed after the selector's training cutoff, in repositories it never saw |
| Primary metric | Share of tasks whose chosen patch passes the task's tests; ties share credit equally | Share whose chosen patch passed tests and was not reverted or reopened within a set window |
| Uncertainty | Paired bootstrap of B minus A, 10,000 resamples, seed 20261027 | The same, plus a power estimate before running |
| Stop rules | Under 100 primary tasks: descriptive only. Over 25 percent of eligible tasks with comments quoting the fix: primary result inconclusive. No changes after the run | Add: stop and report if arm B is worse beyond its interval on the development set |
What a reproducible public pilot found
In the pilot, adding issue history barely changed which patch a simple selector chose, and did not change how often the chosen patch passed. The selector scores each candidate by how many distinct identifiers in its diff, such as function, class and file names, also appear in the text it reads, and picks the highest score. It is crude on purpose: it can be rerun by anyone, has no parameters to tune and cannot have seen the answers.
The pool holds 35,725 candidate patches from 73 submissions, a median of 72 per task. Seventy tasks were solved by no candidate, none by every candidate, and 430 split the pool. Of the 280 tasks that both split the pool and carry comments, 43 (15.4 percent) have comments quoting at least one line of the reference fix, which leaves 237 tasks in the primary set.
On those 237 tasks, the patch chosen from issue text alone passed 59.81 percent of the time and the patch chosen with the comments added passed 60.06 percent of the time, a 0.25-point difference, with a 95 percent bootstrap interval from -2.2 to 2.8 points. On 32 tasks the two arms differed in the pass value of their pick or in the number of candidates tied for the top score; counting any change in which candidates were picked, they differed on 49. Arm B did better on 8 and worse on 8. For scale, a random pick passed 47.2 percent of the time and the single strongest submission in the pool passed 74.3 percent. The held-out slice of 75 tasks falls under the 100-task floor, so it is reported descriptively: both arms reached 56.2 percent.
The pilot results, one run, all figures from the same script:
| Population | Tasks | Issue text only | Plus pre-fix comments | Difference (95% interval) |
|---|---|---|---|---|
| Primary: comments present, pool split, no quoted fix | 237 | 59.81% | 60.06% | 0.25 points (-2.2 to 2.8) |
| Primary, created before 2022 | 162 | 61.5% | 61.8% | 0.4 points (-2.6 to 3.4) |
| Primary, created 2022 or later (held-out, descriptive) | 75 | 56.2% | 56.2% | 0.0 points (-4.0 to 4.0) |
| Comments quote a line of the fix | 43 | 48.8% | 55.0% | 6.2 points (-0.8 to 14.0) |
| All eligible, including quoted fixes | 280 | 58.1% | 59.3% | 1.2 points (-1.1 to 3.6) |
Where the history helped: comments that already quote the fix
The only visible gain came from comments that already contained part of the answer. On the 43 tasks whose comments quote at least one 20-character line of the reference fix, adding the comments moved the chosen patch's pass rate from 48.8 to 55.0 percent. The interval, from -0.8 to 14.0 points, crosses zero, so this is a lead to test rather than a finding.
It is still the pattern a buyer should expect. Issue threads often carry a maintainer's sketch of the fix, a pasted traceback pointing at the exact function, or a link to a draft patch. A selector that reads those comments is not learning to judge patches; it is matching the patch to a hint. In a real deployment that may be exactly what you want, since engineers do read the thread, but an evaluation has to report the two cases apart or it will credit judgment for what was copying.
The pre-fix cutoff matters for the same reason. For GitHub issues, SWE-bench's collector compares each comment's last update with the first commit of the fixing pull request, so a comment edited after work began is dropped even if it was first written earlier. A company export should keep both timestamps and apply the stricter one.
The strongest alternative explanation: a passing test is not a solved problem
The strongest objection to any patch selection result is that passing tests is not the same as solving the problem the issue described. A patch can satisfy the tests written for the fix and still miss the case the reporter cared about, add a regression the tests do not cover, or be reverted a week later.
SWE-bench's evaluation guide counts an instance as resolved when the patch made the required tests pass: the fail-to-pass tests listed for the task, which the fix should turn green, and the pass-to-pass tests, which must stay green. That is a strong, cheap label, and it is the only one the public pilot could use. It cannot see whether the maintainers would have accepted the patch, and it treats two passing patches as equally good when one is a two-line fix and the other rewrites a module.
Private engineering history can do better. A full run should label each candidate with the test result and with what happened after merge: reverted, reopened, followed by a related bug, or held. The primary metric in the design table uses that combined label, which is why a buyer needs issue, merge and incident records linked, not just code.
The engineering decision trace for each experiment task
Each task in the experiment should be stored as one record that a reviewer can rerun. The fields below cover what the pilot used and what a licensed run adds; any field a supplier cannot fill is a reason to narrow the claim, not to guess.
| Field | What it holds | In the public pilot | In a licensed run |
|---|---|---|---|
| Task statement | The issue as first filed, with its timestamp | problem_statement.md | Ticket or issue body at creation, from history |
| Issue history | Comments before the first fix commit, each with created and updated times | hints.md (comments dated before the first fix commit) | Ticket comments, linked chat summaries, review threads, cut at the first fix commit |
| Candidate pool | Every candidate patch and where it came from, generated without the history | 73 leaderboard submissions, 35,725 non-empty patches | Review revisions, abandoned pull requests, agent attempts |
| Test outcome | Pass or fail on the task's tests | Submission results files | CI runs on the candidate against a pinned environment |
| Outcome after merge | Reverted, reopened, related bug, or held, within a stated window | Not available | Revert commits, reopen events, linked incidents |
| Leak flag | Whether the history quotes the fix | Any 20-character line of the reference fix found in hints.md | The same check against the merged fix and draft patches |
| Selection record | Each arm's scores, ties and pick, with seed and selector version | Saved by script for both arms | The same, plus model version, prompt and token counts |
| Permission scope | Which threads may be used, and which must be excluded | Public data | Rights and privacy review of comments, customer names and security threads |
The model-judge arm: designed here, results not published
The arm that matters most for buyers, a language model reading the issue and the candidate diffs and choosing one, has not been run, and no result for it is published here. The design is ready: the same 237-task primary set, the same pool, the model shown candidates in an order shuffled with a published seed, once with issue text and once with the comments added, blind to all test results.
Three rules keep that run honest. Freeze the prompt and model version on the development slice before touching held-out tasks. Report the quoted-fix tasks separately, as the pilot does. And report cost: reading comments adds tokens to every comparison, so a small gain may not pay for itself at the scale of a coding agent's daily runs.
The pilot's interval also sets expectations for size. With 237 paired tasks it spans about 5 points, so a model arm on this set cannot distinguish effects much smaller than that. A buyer who wants to detect a 1 or 2 point gain needs a larger pool of issues with history, which is one reason private archives matter.
How a buyer would run the experiment on licensed engineering history
On licensed data the experiment needs three things a public benchmark lacks: issue threads with full timestamps, more than one realistic candidate per issue, and outcomes that reach past the test run. Review revisions are natural candidates, since each pushed revision is an alternative fix the team considered, and abandoned pull requests for the same issue add rejected options.
Ask a supplier to state how many issues have two or more candidate fixes, how many carry comments before the first fix commit, and what share of those comments quote code, using the same checks as the pilot. Those three counts decide whether the experiment can reach the 100-task floor and whether its result will mean judgment or copying.
SourceX does not keep engineering archives on a shelf; a buyer describes the need through the request form, covering scope, volume, language, format, timing and licensing, and the dataset is sourced from companies willing to license it. The design table and decision trace above can be attached as the acceptance test. Comments often name customers and colleagues, so the owning company's rights and privacy review comes before anything is shared.
Methods note: data versions, selector rules and the single run
Data: SWE-bench/swe-bench-tasks at commit 3d07b46 (August 16, 2026), read for tasks/*/task.yaml, problem_statement.md, hints.md and gold.patch; a task is Verified when its task.yaml lists SWE-bench/SWE-bench_Verified (500 tasks). Candidates: every evaluation/verified/*/all_preds.jsonl or preds.json deleted by swe-bench/experiments commit a3ed594 (Move predictions to s3 bucket, May 21, 2025), read at its parent 05ad61f: 75 files. Labels: each submission's results/results.json resolved list at experiments commit 40f164d (September 3, 2026). No repository code was run; the scripts are the writer's own.
Pool rules: a submission is dropped whole if any Verified task appears more than once in its file (2 dropped); a candidate is a record with a non-empty patch after removing whitespace (503 empty dropped). Tokens: the regex [A-Za-z_][A-Za-z0-9_]*, lowercased, at least 4 characters, Python keywords removed. A patch's identifiers are the tokens on changed lines (starting with + or - but not +++ or ---) and on diff --git path lines. Score: distinct patch identifiers that also occur in the context. Ties share the pick equally, so a task's value is the passing fraction among tied top candidates, kept as an exact fraction.
Eligible: hints.md non-empty after stripping whitespace and the pool holds at least one passing and one failing candidate. Leak flag: a line added by gold.patch (starting with + but not +++), stripped, 20 or more characters long, found verbatim in hints.md. Held-out: task.yaml created_at on or after 2022-01-01. The bootstrap resamples tasks sorted by instance id, 10,000 times; each resample draws one task index per task with randrange on a single Python random.Random(20261027) generator, and the interval is the 250th and 9,750th sorted mean differences. The interval reproduces only with that sampler; another sampler, such as random.choices with the same seed, gives slightly different bounds. Differences are computed from exact fractions, so they can differ by 0.1 from the gap between two rounded rates. Percentages are computed from exact fractions and rounded once. The best single submission is the one with the most passing tasks across all 500 (folder 20250514_aime_coder).
The script was written and frozen before its first run and was run once; nothing was tuned. Not measured: model-based selectors, comments after the first fix commit, review discussion, and any outcome after merge. Scripts and outputs (sb_load.py, sb_pilot.py, sb_pilot_out.json, sb_pilot_stdout.txt) are kept in the writer's scratch folder; saved outputs hold task ids, counts and fractions, no comment text or names.
Frequently asked questions
What is the hints field in SWE-bench?
It holds discussion from the issue, comments posted before the first commit of the pull request that fixed it, kept in hints.md in the task data. SWE-bench's submission checklist asks leaderboard entrants not to use it, so public results say nothing about its value.
Can issue comments leak the answer into a patch selection test?
Yes. In the public pilot, 43 of 280 eligible SWE-bench Verified tasks had comments quoting at least one 20-character line of the reference fix. Report those tasks separately, or a selector gets credit for matching a hint instead of judging the patch.
How many tasks does an issue history experiment need?
More than it looks. With 237 paired tasks the pilot's 95 percent interval spans about 5 points, so smaller effects are invisible. Set a floor before running; the pilot used 100 tasks as the minimum for any conclusion.
Why test a simple lexical selector before a language model?
Because it is reproducible, has nothing to tune and cannot have memorized the benchmark. It shows whether the history carries usable signal at all and sizes the leak problem. It does not stand in for a model judge, whose result is not published here.
Does a patch that passes the tests solve the issue?
Not necessarily. Tests written for the fix can pass while the reporter's real case stays broken or a regression appears elsewhere. A full experiment on company data should also label candidates by what happened after merge, such as a revert or a reopened ticket.
Sources
- SWE-bench's documentation lists problem_statement.md as the issue text shown to a model and hints.md as discussion from the issue, absent when there is none. Source
- SWE-bench's dataset guide lists SWE-bench Verified as expert-verified solvable problems, 500 instances. Source
- SWE-bench's collector keeps GitHub issue comments last updated, and Django ticket comments timestamped, before the first commit of the fixing pull request. Source
- SWE-bench's submission checklist asks entrants to confirm they do not use the hints field and defines Best@k selection by a distinct module without the benchmark's tests. Source
- The SWE-bench task data at commit 3d07b46 supplies the 500 Verified tasks, their comments and reference fixes used in the pilot. Source
- Leaderboard prediction files for SWE-bench Verified at experiments commit 05ad61f supply the candidate patches. Source
- Each submission's results file at experiments commit 40f164d supplies which tasks it resolved. Source
- SWE-bench's evaluation guide counts an instance as resolved when the patch made the required tests pass. Source
- Agentless re-ranks candidate patches using regression and reproduction test results and selects one to submit. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.