AI uses for records
Failed test design decision dataset: what 2,219 slowdowns changed
By SourceX Editorial · Updated
10 sections
Short answer
A failed test design decision dataset pairs each negative result with the decision it forced, the change that followed and the record that explains it. Rust's public performance logs show how thin that chain gets: of 2,219 benchmark regressions logged from 2020 to 2026, at least 48 are recorded as withdrawn or redone, 36.5 percent record no decision, and 12 of the 44 merge commits behind those changes give no performance reason.
Evaluation split
Failed test design decision
Key takeaways
- A negative result teaches a model something only when the record keeps the result, the decision, the change and the link between them; most public records keep one or two of the four.
- In the rejected-option sections of 318 Python design proposals, 9 of 5,987 text blocks report an observed test or measurement that counted against an option, and 3 give the numbers.
- Rust's weekly performance triage logged 2,219 regressions from July 2020 to October 2026; 811 record no decision, and at least 48 are recorded as withdrawn or redone.
- Notes moved from bullets to prose rather than thinning: 70.5 percent of 2026's entries have no bullet note, yet only one has no note at all, and 36.4 percent give a reason for keeping the slowdown, against 27.4 percent in 2022.
- On the change side, 32 of 44 merge commits state the slowdown; 4 are bare reverts, and one says it does not affect performance although the log credits it with the fix.
What belongs in a failed test design decision dataset?
Four linked records per case: the negative result, meaning what was measured, against which baseline and by how much; the decision it prompted, whether to withdraw, redo, keep or dismiss it as noise; the change that carried the decision out; and the written reason that ties them together. A failed test on its own is a log line. The supervision sits in the decision and the link.
The decision here belongs to an AI team buying engineering data: whether a supplier's history can teach a coding agent when a negative result should change a design, and can grade an agent facing the same choice. Merged code shows only the path that survived. Abandoned approaches and documented corrections sit elsewhere, and they decay unless someone keeps them.
We counted two public record types that ought to hold such cases. Python's design proposals must list rejected ideas, and Rust's compiler performance team logs every benchmark regression after merge. Then we checked whether the changes that followed say why in their own commit messages.
Where negative results get written, and where they get lost
In three places that rarely point at one another: the design document, where an option is ruled out; the measurement log, where a benchmark or test result is recorded; and the change record, where code is reverted or rewritten. Each place keeps a different part of the case.
Design documents keep the alternatives and the argument. Measurement logs keep the number and, sometimes, a note on what to do about it. Change records keep the code and whatever its author chose to write. A buyer who licenses only one of the three gets cases with a result and no decision, or a decision and no result.
Survival differs as well. Text inside a repository travels with every clone, while benchmark comparisons usually live on a dashboard, test logs on a CI service with its own retention rules and review threads on a code host. A link from the repository is only as durable as the system it points to, so the counts below measure what each public record keeps inside the repository itself.
Do design documents keep the test that ruled an option out?
Rarely. In the rejected-option sections of Python Enhancement Proposals, 9 of 5,987 text blocks report an observed test or measurement that counted against an option, and only 3 of those 9 give the numbers.
PEPs are a fair test because the process asks for exactly this record. PEP 1 lists rejected ideas among the parts of a successful proposal and says 'Those rejected ideas should be recorded along with the reasoning as to why they were rejected.' At commit 35d53a8 of the python/peps repository, dated 27 October 2026, 318 of 746 PEPs have a section whose heading names rejected or alternative options.
We searched those sections for measurement words such as benchmark, measured, experiment, prototype and timeit, found 32 blocks in 23 PEPs, and read every one. Seventeen used the word in another sense, and two argued that an option would be slower without measuring it. Four more described a prototype without a measured verdict, offered a benchmark nobody had run, or measured something that ruled nothing out. Nine reported an observed result against an option. The best keep the evidence inline: PEP 709 records that an alternate approach 'provides roughly half of the performance benefit', a 1.5x gain on a microbenchmark and 4 percent on a pyperformance benchmark, and links the pull request that built it. PEP 509 pastes the timeit runs behind its claim that checking a dictionary's version at Python level is not faster than a lookup: 47.5 against 48.7 nanoseconds.
Most measured rejections are one sentence with no number and no pointer, such as PEP 523's note that, after measurements, losing a field was 'deemed too costly to performance'. A wider net that adds words like slower, tried and tested catches 126 blocks in 81 PEPs; we did not label those, so 9 is a floor. The finding concerns location rather than quality: the measurements behind design choices mostly live in pull requests and benchmark threads, outside the document that records the choice.
What 2,219 Rust benchmark regressions record about the decision that followed
The regression is always recorded; a decision often is not. Across 326 weekly triage logs from July 2020 to October 2026 we found 2,219 regression or mixed entries. Only 66 carry no note at all, but 811, 36.5 percent, record no decision: no note, or a note that stops short of one.
The process expects follow-through. The README in the rust-lang/rustc-perf repository tells each week's triager to check the previous log, 'Look for responses in PRs, and follow up on any promised actions.' Each entry carries the measured change as a summary table and links a comparison page on the project's performance dashboard for detail. Rollups, merges that bundle several pull requests, account for 669 entries, so a triager often has to find the culprit before any decision is possible.
Notes have changed form rather than thinned. Since 2022 triagers have increasingly written a prose paragraph under an entry instead of bullets: 70.5 percent of 2026's entries have no bullet note, yet only one has no note at all, and the share giving a reason for keeping a slowdown rose from 27.4 percent of 2022's entries to 36.4 percent of 2026's. The classes below come from one person's reading of the 142 entries whose bullet notes contain a change word, such as revert, fixed in or follow-up, and from fixed keyword rules for the rest; 81 entries that mention a change only in prose were not read, so the withdrawn and redone counts are floors:
| Outcome class | Entries | Share of 2,219 | What the record shows |
|---|---|---|---|
| Withdrawn | 21 | 0.9% | The change, or its culprit, was reverted |
| Redone in a follow-up | 27 | 1.2% | A later pull request is named as fixing or recovering the slowdown |
| Change proposed, not confirmed | 47 | 2.1% | A revert suggested or a fix pending, with no outcome in that entry |
| Kept, reason stated | 670 | 30.2% | Justified, expected, a correctness fix, or gains outweigh losses |
| Judged noise | 393 | 17.7% | Benchmark noise, a bimodal result or a spurious blip |
| Still open | 156 | 7.0% | An investigation or a ping with no close |
| Entry is itself a revert or fix | 13 | 0.6% | The regressing change undoes or repairs an earlier one |
| Change mentioned in prose, not read | 81 | 3.7% | A revert, fix or follow-up appears only in prose; unlabeled |
| No decision recorded | 811 | 36.5% | No note (66) or a note with no decision (745) |
Which slowdowns changed the design, and does the change say why?
At least 48 entries record a changed design, 21 withdrawn and 27 redone. The link from the log to the change is usually present, and the reason on the change side is usually present too, though not always.
On the log side, 44 of the 48 notes name another pull request, typically the revert or the fix, while 4 withdrawals name none. Only 10 of the 21 withdrawals state the slowdown as the reason; the others record that a revert happened, which a reader can connect to the slowdown but cannot confirm from the log.
On the change side, we found on Rust's main branch the merge commit of every revert or fix the logs identify, 44 distinct commits, and read each message. Thirty-two give the slowdown, or a benchmark it broke, as a reason; the clearest says it is 'Reverting per performance regression noted post-merge' and links the comment where the regression surfaced. Twelve give no performance reason. Four are bare reverts that say only which change they undo, and one fix, credited in a February 2024 log with resolving a regression, says of itself 'So this PR does not affect perf, but could still be considered a cleanup'. Records that disagree like this are labels to adjudicate, not to copy.
Some of the most useful cases are negative results about the correction itself. In June 2021 a triager noted that a revert made for performance concerns 'does not seem to have recovered performance', and suggested re-landing the original with measurements. In November 2023 another wrote that follow-up pull requests had been proposed, 'but we have not successfully found one that undoes the regression.' A dataset that keeps only successful fixes drops exactly these.
The decision trace: fields a negative-result record needs
A usable case lets a second reader see the result, the decision, who made it, the change and whether the change worked, each with a date. The Rust records show which of those fields a public history already holds and which go missing.
Two fields decide whether a label can be trusted: the decision, because silence is not a decision, and verification, because a revert that did not recover the metric teaches the opposite lesson. The trace, field by field, with what the Rust logs and merge commits supply:
| Field | What it holds | In the Rust records | Acceptance check |
|---|---|---|---|
| Negative result | Metric, benchmark set, size of the change and baseline | A summary table in every entry; detail behind a dashboard link | Numbers stored in the record, not only behind a link |
| Change under test | The commit or pull request measured | Linked in every entry; 669 entries are rollups | One change per case, with the culprit found inside bundles |
| Decision | Withdraw, redo, keep or dismiss as noise | 811 of 2,219 entries record none | A label from a fixed set, never inferred from silence |
| Stated reason | Why the result did or did not change the design | 670 entries give a reason for keeping | Reason text kept verbatim with its date |
| Link to the change | The pull request or commit that carried out the decision | 44 of 48 changed entries name one | Resolves to a merged commit |
| Change record's own reason | What the revert or fix says about the result | 32 of 44 merge messages state the slowdown | Disagreements flagged, not settled by guesswork |
| Verification | Whether the change recovered the metric | Only in later notes, when at all | A later measurement linked, or the case marked unverified |
| Decider | Who decided, recorded as a role | Triager identified per log; authors and reviewers in threads | Role kept, personal names removed |
Outcome rubric: seven labels for what a negative result did
Seven labels cover the cases these records produce, and only two of them mean the result changed the design. Each label requires specific evidence, and a case that lacks it takes the weaker label.
Keeping the counterexamples matters as much as keeping the reversals. The labels, with the evidence each needs and how many Rust entries earn it:
| Label | Evidence required | Rust entries | Use in training or evaluation |
|---|---|---|---|
| Withdrawn | A merged revert, with the reason tied to the result | 21, of which 10 state the reason | Positive examples of stopping a change on evidence |
| Redone | A later change named as recovering the metric | 27 | Pairs of a failed approach and its replacement |
| Kept with a reason | A written reason for accepting the cost | 670 | Counterexamples where the design should stand |
| Dismissed as noise | A stated noise judgment, ideally with a rerun | 393 | Teaches checking a result before acting on it |
| Proposed, unconfirmed | A suggested revert or fix with no outcome | 47 | Held out until the outcome is found |
| Open | An investigation or ping that never closed | 156 | Excluded from outcome labels |
| Undocumented | No note, or a note with no decision | 811 | Usable only as unlabeled results |
The strongest objection: a recovered benchmark is not a solved problem
True, and the records show why. A benchmark stands in for what users experience, so a withdrawn change may have protected nobody, and a kept regression may have been the right call: 670 entries accept the cost for a stated reason, often a correctness or soundness fix.
Three cautions follow for anyone labeling these records. Noise is common, 393 entries, so a negative result needs a rerun before it counts. Rollups blur attribution, so a culprit named in a note is a hypothesis until a targeted run confirms it. And a triage note is one person's reading; when the merge message says something else, as with the fix that says it does not affect performance, the record holds a disagreement, not an answer.
The workable response is to label the decision, not its wisdom. Whether withdrawing a change was right is often unknowable from the record. Whether the record shows a result, a decision, a reason and a link is checkable, and that is what a model can learn from.
How buyers would use these records, and what company archives add
As decision examples for training and as held-out choices for evaluation. Give an agent the measured slowdown and the change, ask whether to revert, redo or keep, and grade it against the recorded decision and reason. Withdrawn and redone cases supply the positives; kept-with-reason cases supply the counterexamples that stop a model from reverting everything. Work on agents that learn from failure, such as Reflexion, whose agents 'verbally reflect on task feedback signals' and store that text in an episodic memory buffer, shows why the written reason matters more than a pass or fail flag.
Company archives hold richer versions of the same cases: load tests that failed a release gate, A/B experiments that ended a feature, spikes abandoned after a prototype and design reviews that cite a benchmark. They can also hold what public projects rarely link, the business outcome. Those records carry more risk, including customer names in incident notes, internal hostnames in test logs and contract limits on licensing, so rights review and de-identification come before anything leaves the company.
SourceX's request page says datasets are sourced on request, not held in stock, and asks a buyer to share scope, volume, language, format, timing and licensing. For negative-result cases, add the intended use, the decision labels you will accept and an acceptance test, for example that every withdrawn or redone case resolves to a merged change and states its reason, and that undocumented results arrive flagged as unlabeled. Whether a given company's history passes that test is settled only by reviewing it.
Methods note: repositories, counting rules and limits
Triage logs: rust-lang/rustc-perf at commit 7c518b0 (27 October 2026), the weekly files named by date in triage/ plus triage/2025 and triage/2026, 326 files from 2020-07-14 to 2026-10-26; triage/2020.md, a compilation of seven earlier logs in another format, was excluded, and the first five logs, which list regressions as bullets without Regressions or Mixed headings, contribute none. One entry is counted per regression: a non-indented title line under a Regressions or Mixed heading that starts a paragraph (or ends a title wrapped over several lines), links a rust-lang/rust pull request (2021 logs link it through /issues/) and carries or is followed by the measured change, a comparison link or summary table. Prose lines that link a culprit, a revert or a comment are notes, not entries. An entry's notes are the bullets and the prose paragraphs written under it, with lazy continuation lines joined. The 142 entries whose bullet notes contain a change word (revert, back out, re-land, follow-up, fixed, resolved, addressed or recovered in or by, undo, undone, mitigated) were read and labeled by one person; the 81 entries with a change word only in prose were not read, so the withdrawn and redone counts are floors. The rest were classed by whole-word keyword lists, words for keeping a slowdown first (justified, expected, acceptable, worth, outweigh, correctness, soundness, anticipated and others), then noise words, then open words; the script prints every list.
Merge commits: a commit-only clone of rust-lang/rust main at 41089d0 (27 October 2026), where a pull request's merge commit is the one whose subject starts 'Auto merge of #N' or 'Rollup merge of #N'; each of the 44 messages was read and marked for whether it gives the slowdown or a broken benchmark as a reason. PEPs: python/peps at 35d53a8, 746 pep-*.rst files; a rejected-option section has a heading with the whole word rejected, alternative, alternatives, abandoned or discarded; blocks are separated by blank lines, and the 32 blocks containing any of 24 measurement words (benchmark, benchmarks, benchmarked, benchmarking, measured, measurement, measurements, measuring, experiment, experiments, experimented, prototype, prototypes, prototyped, profiling, profiled, timings, timeit, pyperformance, pyperf, slowdown, speedup, test suite, crater) were read and labeled; PEP 744's note that early prototypes showed too little optimization potential is a borderline case counted among the nine. Shares come from integers, rounded once, half up. Not measured: review threads, dashboard data, whether withdrawn changes later re-landed, and any company's internal records. Saved files hold pull request numbers, dates and labels, with no names or message text.
Frequently asked questions
What is a failed test design decision dataset?
A set of cases that each pair a negative test or benchmark result with the decision it prompted, the change that carried the decision out and the written reason linking them. It teaches a model when a failing result should change a design, and when it should not.
Why keep negative results that did not change the design?
Because a model trained only on reversals learns to revert everything. In Rust's triage logs, 670 regressions were kept for a stated reason, such as a correctness fix, against at least 48 that led to a withdrawal or a rewrite. Both kinds belong in the data.
Do design documents record failed experiments?
Seldom in detail. In 318 Python Enhancement Proposals with rejected-option sections, 9 of 5,987 text blocks report an observed test or measurement against an option, and 3 give numbers. The measurements mostly live in pull requests and benchmark threads.
Is a revert commit enough to label a failed approach?
No. Four of the 44 Rust merge commits that carried out a decision are bare reverts with no reason, and one fix says it does not affect performance although the triage log credits it. Pair each revert with the measurement and the note that explains it.
What should a buyer ask a supplier about negative-result records?
Ask for denominators: how many logged failures carry a decision, how many decisions link to a merged change, and how many changes state the failing result as their reason. Ask too whether dashboard and CI detail is exported with the repository or left behind.
Sources
- Rust's weekly compiler performance triage logs, July 2020 to October 2026, hold 2,219 regression or mixed entries; 811 record no decision, at least 48 record a withdrawal or a redo, and a November 2023 note records no follow-up yet undoing a regression. Source
- Each week's triager is told to check the previous log, look for responses in pull requests and follow up on promised actions. Source
- A June 2021 note records that a revert made for performance concerns did not seem to recover performance. Source
- The merge commits for all 44 pull requests named as reverts or fixes are on Rust's main branch; 32 give the slowdown or a broken benchmark as a reason. Source
- A 2021 revert's merge message says it reverts per a performance regression noted after merge and links where it was found. Source
- A 2024 merge message says the change does not affect performance, though the triage log credits it with resolving a regression. Source
- PEP 1 asks that rejected ideas be recorded with the reasoning for rejecting them. Source
- Of 746 PEPs, 318 have a rejected-option section; 9 of their 5,987 blocks report an observed test or measurement against an option, among them PEP 509's timeit runs and PEP 523's unquantified note. Source
- PEP 709 records that an alternate approach gave roughly half the performance benefit and links the pull request that built it. Source
- Reflexion agents reflect in words on task feedback and keep that reflective text in an episodic memory buffer. Source
- SourceX sources datasets on request rather than holding them in stock, and the request form asks for scope, volume, language, format, timing and licensing. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.