Skip to content

AI uses for records

Issue history code patch experiment: a design and a 237-task pilot

By SourceX Editorial · Updated

10 sections

Short answer

An issue history code patch experiment holds the candidate patches fixed and changes one thing: whether the selector also reads the issue's pre-fix comments. In a public pilot on 237 SWE-bench Verified tasks, adding those comments to a lexical selector moved the chosen patch's pass rate from 59.81 to 60.06 percent, a 0.25-point difference with an interval from -2.2 to 2.8 points. Model-based selectors were not run.

Record timeline

Issue history code patch

This page covers issue history code patch experiment: a design and a 237-task pilot.Illustrative

Key takeaways

  • Hold the candidate pool fixed and vary only what the selector reads: issue text alone, or issue text plus comments posted before the fix began.
  • Public leaderboards cannot answer the question, because SWE-bench asks entrants not to use the hints field that holds those comments.
  • In a frozen, single-run pilot on 237 tasks, comments moved a lexical selector's pass rate by 0.25 points, with an interval from -2.2 to 2.8.
  • The visible gain sat in the 43 tasks whose comments quote the fix, so leak checks have to be reported apart from judgment.
  • A full run on licensed history should label candidates by what happened after merge, not only by tests.

What does an issue history code patch experiment test?

An issue history code patch experiment tests one claim: that a system choosing among candidate patches for the same issue picks a working patch more often when it can read the issue's discussion history, not just the issue text. Everything else is held fixed, including the candidates, the selector and the grading, so any difference comes from the history alone.

Patch selection is a real stage in coding agents. The Agentless project's README describes generating several candidate patches, running regression tests and a generated reproduction test, and using the results to re-rank the remaining patches and pick one to submit. SWE-bench's own submission checklist allows a Best@k system, in which a distinct module of the system decides which attempt to submit without using the benchmark's evaluation or its fail-to-pass and pass-to-pass tests.

For a buyer the decision is practical: whether to pay for issue threads, review discussion and ticket comments alongside code, or whether issue titles and descriptions are enough. This page publishes the design for answering that, runs the one arm that can be run reproducibly on public data, and states plainly what has not been run.

Why public leaderboards leave the question open

Public leaderboards cannot settle it because their rules keep issue history out of the systems they rank. SWE-bench's submission checklist asks entrants to confirm that the system does not use the hints field, and its note on test use repeats that a system must not use the pass-to-pass tests, the fail-to-pass tests or the hints_text field.

The history exists in the benchmark, though. SWE-bench's documentation describes each task directory as holding problem_statement.md, the issue text shown to a model, and hints.md, discussion from the issue, absent when there is none. The collection code keeps only comments from before the fix began: it takes the time of the first commit in the fixing pull request and keeps GitHub issue comments whose last update came earlier, or, for Django's Trac tickets, ticket comments timestamped earlier.

That makes a clean experiment possible. Every candidate on the leaderboard was generated without the comments, so the comments can be added at selection time without having shaped the candidates. In the public task data, 335 of the 500 SWE-bench Verified tasks carry non-empty comments.

The experiment design: hypothesis, held-out set, comparison, metric and stop rules

The design fixes seven elements before any result is seen. The middle column is what the public pilot fixed and applied; the right column is what a full run on licensed company data should use, where the candidates, the history and the outcome labels are richer than a public benchmark's.

The experiment design: hypothesis, held-out set, comparison, metric and stop rules
ElementFixed for the public pilotFor a full run on licensed history
HypothesisWith the candidate pool held fixed, a selector reading issue text plus pre-fix comments picks a test-passing patch more often than the same selector reading issue text aloneThe same, stated for each selector before any run
Unit and poolA SWE-bench Verified task with comments and at least one passing and one failing candidate; every non-empty patch from 73 leaderboard submissionsAn issue with two or more candidate fixes: review revisions, abandoned pull requests, agent outputs
ComparisonArm A reads problem_statement.md; arm B reads it plus hints.md; same scorer, same poolSame two arms; candidate order shuffled with a published seed for model selectors
Held-out setTasks created on or after January 1, 2022 (75 in the primary set), reported separatelyIssues filed after the selector's training cutoff, in repositories it never saw
Primary metricShare of tasks whose chosen patch passes the task's tests; ties share credit equallyShare whose chosen patch passed tests and was not reverted or reopened within a set window
UncertaintyPaired bootstrap of B minus A, 10,000 resamples, seed 20261027The same, plus a power estimate before running
Stop rulesUnder 100 primary tasks: descriptive only. Over 25 percent of eligible tasks with comments quoting the fix: primary result inconclusive. No changes after the runAdd: stop and report if arm B is worse beyond its interval on the development set

What a reproducible public pilot found

In the pilot, adding issue history barely changed which patch a simple selector chose, and did not change how often the chosen patch passed. The selector scores each candidate by how many distinct identifiers in its diff, such as function, class and file names, also appear in the text it reads, and picks the highest score. It is crude on purpose: it can be rerun by anyone, has no parameters to tune and cannot have seen the answers.

The pool holds 35,725 candidate patches from 73 submissions, a median of 72 per task. Seventy tasks were solved by no candidate, none by every candidate, and 430 split the pool. Of the 280 tasks that both split the pool and carry comments, 43 (15.4 percent) have comments quoting at least one line of the reference fix, which leaves 237 tasks in the primary set.

On those 237 tasks, the patch chosen from issue text alone passed 59.81 percent of the time and the patch chosen with the comments added passed 60.06 percent of the time, a 0.25-point difference, with a 95 percent bootstrap interval from -2.2 to 2.8 points. On 32 tasks the two arms differed in the pass value of their pick or in the number of candidates tied for the top score; counting any change in which candidates were picked, they differed on 49. Arm B did better on 8 and worse on 8. For scale, a random pick passed 47.2 percent of the time and the single strongest submission in the pool passed 74.3 percent. The held-out slice of 75 tasks falls under the 100-task floor, so it is reported descriptively: both arms reached 56.2 percent.

The pilot results, one run, all figures from the same script:

What a reproducible public pilot found
PopulationTasksIssue text onlyPlus pre-fix commentsDifference (95% interval)
Primary: comments present, pool split, no quoted fix23759.81%60.06%0.25 points (-2.2 to 2.8)
Primary, created before 202216261.5%61.8%0.4 points (-2.6 to 3.4)
Primary, created 2022 or later (held-out, descriptive)7556.2%56.2%0.0 points (-4.0 to 4.0)
Comments quote a line of the fix4348.8%55.0%6.2 points (-0.8 to 14.0)
All eligible, including quoted fixes28058.1%59.3%1.2 points (-1.1 to 3.6)
Left, a funnel from 500 SWE-bench Verified tasks to 335 with pre-fix comments, 280 eligible and 237 in the primary set; right, bars of how often the chosen patch passed: random pick 47.2 percent, issue text only 59.8, plus comments 60.1, best single submission 74.3, perfect choice 100.
Public pilot, one frozen run of a lexical selector. Source: SWE-bench task data and swe-bench/experiments submissions, counted October 27, 2026.

Where the history helped: comments that already quote the fix

The only visible gain came from comments that already contained part of the answer. On the 43 tasks whose comments quote at least one 20-character line of the reference fix, adding the comments moved the chosen patch's pass rate from 48.8 to 55.0 percent. The interval, from -0.8 to 14.0 points, crosses zero, so this is a lead to test rather than a finding.

It is still the pattern a buyer should expect. Issue threads often carry a maintainer's sketch of the fix, a pasted traceback pointing at the exact function, or a link to a draft patch. A selector that reads those comments is not learning to judge patches; it is matching the patch to a hint. In a real deployment that may be exactly what you want, since engineers do read the thread, but an evaluation has to report the two cases apart or it will credit judgment for what was copying.

The pre-fix cutoff matters for the same reason. For GitHub issues, SWE-bench's collector compares each comment's last update with the first commit of the fixing pull request, so a comment edited after work began is dropped even if it was first written earlier. A company export should keep both timestamps and apply the stricter one.

The strongest alternative explanation: a passing test is not a solved problem

The strongest objection to any patch selection result is that passing tests is not the same as solving the problem the issue described. A patch can satisfy the tests written for the fix and still miss the case the reporter cared about, add a regression the tests do not cover, or be reverted a week later.

SWE-bench's evaluation guide counts an instance as resolved when the patch made the required tests pass: the fail-to-pass tests listed for the task, which the fix should turn green, and the pass-to-pass tests, which must stay green. That is a strong, cheap label, and it is the only one the public pilot could use. It cannot see whether the maintainers would have accepted the patch, and it treats two passing patches as equally good when one is a two-line fix and the other rewrites a module.

Private engineering history can do better. A full run should label each candidate with the test result and with what happened after merge: reverted, reopened, followed by a related bug, or held. The primary metric in the design table uses that combined label, which is why a buyer needs issue, merge and incident records linked, not just code.

The engineering decision trace for each experiment task

Each task in the experiment should be stored as one record that a reviewer can rerun. The fields below cover what the pilot used and what a licensed run adds; any field a supplier cannot fill is a reason to narrow the claim, not to guess.

The engineering decision trace for each experiment task
FieldWhat it holdsIn the public pilotIn a licensed run
Task statementThe issue as first filed, with its timestampproblem_statement.mdTicket or issue body at creation, from history
Issue historyComments before the first fix commit, each with created and updated timeshints.md (comments dated before the first fix commit)Ticket comments, linked chat summaries, review threads, cut at the first fix commit
Candidate poolEvery candidate patch and where it came from, generated without the history73 leaderboard submissions, 35,725 non-empty patchesReview revisions, abandoned pull requests, agent attempts
Test outcomePass or fail on the task's testsSubmission results filesCI runs on the candidate against a pinned environment
Outcome after mergeReverted, reopened, related bug, or held, within a stated windowNot availableRevert commits, reopen events, linked incidents
Leak flagWhether the history quotes the fixAny 20-character line of the reference fix found in hints.mdThe same check against the merged fix and draft patches
Selection recordEach arm's scores, ties and pick, with seed and selector versionSaved by script for both armsThe same, plus model version, prompt and token counts
Permission scopeWhich threads may be used, and which must be excludedPublic dataRights and privacy review of comments, customer names and security threads

The model-judge arm: designed here, results not published

The arm that matters most for buyers, a language model reading the issue and the candidate diffs and choosing one, has not been run, and no result for it is published here. The design is ready: the same 237-task primary set, the same pool, the model shown candidates in an order shuffled with a published seed, once with issue text and once with the comments added, blind to all test results.

Three rules keep that run honest. Freeze the prompt and model version on the development slice before touching held-out tasks. Report the quoted-fix tasks separately, as the pilot does. And report cost: reading comments adds tokens to every comparison, so a small gain may not pay for itself at the scale of a coding agent's daily runs.

The pilot's interval also sets expectations for size. With 237 paired tasks it spans about 5 points, so a model arm on this set cannot distinguish effects much smaller than that. A buyer who wants to detect a 1 or 2 point gain needs a larger pool of issues with history, which is one reason private archives matter.

How a buyer would run the experiment on licensed engineering history

On licensed data the experiment needs three things a public benchmark lacks: issue threads with full timestamps, more than one realistic candidate per issue, and outcomes that reach past the test run. Review revisions are natural candidates, since each pushed revision is an alternative fix the team considered, and abandoned pull requests for the same issue add rejected options.

Ask a supplier to state how many issues have two or more candidate fixes, how many carry comments before the first fix commit, and what share of those comments quote code, using the same checks as the pilot. Those three counts decide whether the experiment can reach the 100-task floor and whether its result will mean judgment or copying.

SourceX does not keep engineering archives on a shelf; a buyer describes the need through the request form, covering scope, volume, language, format, timing and licensing, and the dataset is sourced from companies willing to license it. The design table and decision trace above can be attached as the acceptance test. Comments often name customers and colleagues, so the owning company's rights and privacy review comes before anything is shared.

Methods note: data versions, selector rules and the single run

Data: SWE-bench/swe-bench-tasks at commit 3d07b46 (August 16, 2026), read for tasks/*/task.yaml, problem_statement.md, hints.md and gold.patch; a task is Verified when its task.yaml lists SWE-bench/SWE-bench_Verified (500 tasks). Candidates: every evaluation/verified/*/all_preds.jsonl or preds.json deleted by swe-bench/experiments commit a3ed594 (Move predictions to s3 bucket, May 21, 2025), read at its parent 05ad61f: 75 files. Labels: each submission's results/results.json resolved list at experiments commit 40f164d (September 3, 2026). No repository code was run; the scripts are the writer's own.

Pool rules: a submission is dropped whole if any Verified task appears more than once in its file (2 dropped); a candidate is a record with a non-empty patch after removing whitespace (503 empty dropped). Tokens: the regex [A-Za-z_][A-Za-z0-9_]*, lowercased, at least 4 characters, Python keywords removed. A patch's identifiers are the tokens on changed lines (starting with + or - but not +++ or ---) and on diff --git path lines. Score: distinct patch identifiers that also occur in the context. Ties share the pick equally, so a task's value is the passing fraction among tied top candidates, kept as an exact fraction.

Eligible: hints.md non-empty after stripping whitespace and the pool holds at least one passing and one failing candidate. Leak flag: a line added by gold.patch (starting with + but not +++), stripped, 20 or more characters long, found verbatim in hints.md. Held-out: task.yaml created_at on or after 2022-01-01. The bootstrap resamples tasks sorted by instance id, 10,000 times; each resample draws one task index per task with randrange on a single Python random.Random(20261027) generator, and the interval is the 250th and 9,750th sorted mean differences. The interval reproduces only with that sampler; another sampler, such as random.choices with the same seed, gives slightly different bounds. Differences are computed from exact fractions, so they can differ by 0.1 from the gap between two rounded rates. Percentages are computed from exact fractions and rounded once. The best single submission is the one with the most passing tasks across all 500 (folder 20250514_aime_coder).

The script was written and frozen before its first run and was run once; nothing was tuned. Not measured: model-based selectors, comments after the first fix commit, review discussion, and any outcome after merge. Scripts and outputs (sb_load.py, sb_pilot.py, sb_pilot_out.json, sb_pilot_stdout.txt) are kept in the writer's scratch folder; saved outputs hold task ids, counts and fractions, no comment text or names.

Frequently asked questions

What is the hints field in SWE-bench?

It holds discussion from the issue, comments posted before the first commit of the pull request that fixed it, kept in hints.md in the task data. SWE-bench's submission checklist asks leaderboard entrants not to use it, so public results say nothing about its value.

Can issue comments leak the answer into a patch selection test?

Yes. In the public pilot, 43 of 280 eligible SWE-bench Verified tasks had comments quoting at least one 20-character line of the reference fix. Report those tasks separately, or a selector gets credit for matching a hint instead of judging the patch.

How many tasks does an issue history experiment need?

More than it looks. With 237 paired tasks the pilot's 95 percent interval spans about 5 points, so smaller effects are invisible. Set a floor before running; the pilot used 100 tasks as the minimum for any conclusion.

Why test a simple lexical selector before a language model?

Because it is reproducible, has nothing to tune and cannot have memorized the benchmark. It shows whether the history carries usable signal at all and sizes the leak problem. It does not stand in for a model judge, whose result is not published here.

Does a patch that passes the tests solve the issue?

Not necessarily. Tests written for the fix can pass while the reporter's real case stays broken or a regression appears elsewhere. A full experiment on company data should also label candidates by what happened after merge, such as a revert or a reopened ticket.

Sources

  • SWE-bench's documentation lists problem_statement.md as the issue text shown to a model and hints.md as discussion from the issue, absent when there is none. Source
  • SWE-bench's dataset guide lists SWE-bench Verified as expert-verified solvable problems, 500 instances. Source
  • SWE-bench's collector keeps GitHub issue comments last updated, and Django ticket comments timestamped, before the first commit of the fixing pull request. Source
  • SWE-bench's submission checklist asks entrants to confirm they do not use the hints field and defines Best@k selection by a distinct module without the benchmark's tests. Source
  • The SWE-bench task data at commit 3d07b46 supplies the 500 Verified tasks, their comments and reference fixes used in the pilot. Source
  • Leaderboard prediction files for SWE-bench Verified at experiments commit 05ad61f supply the candidate patches. Source
  • Each submission's results file at experiments commit 40f164d supplies which tasks it resolved. Source
  • SWE-bench's evaluation guide counts an instance as resolved when the patch made the required tests pass. Source
  • Agentless re-ranks candidate patches using regression and reproduction test results and selects one to submit. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies