AI uses for records
Software test false confidence dataset: tests that shared the bug
By SourceX Editorial · Updated
10 sections
Short answer
Compare each test's expected values with the requirement, not with the code. Tests that share the code's wrong assumption usually compute expected results the same way the code does and get edited in the commit that fixes the bug. Confirm each case with a test written from the requirement alone that fails on the buggy revision and passes on the fixed one, linked to review and production evidence.
License terms
Software test false
Key takeaways
- A passing test can encode the same misreading of a requirement as the code it checks.
- Coverage and mutation scores measure consistency with the code, not agreement with the requirement.
- The strongest history signal is a fix commit that changes a test's expected values along with the code.
- Each case needs a requirement grounded test that fails on the buggy revision and passes on the fixed one.
- Rule out changed requirements, missing tests and flaky tests before labeling a case.
- A merged pull request is not proof of outcome; state the post-deployment evidence and its window.
How do you detect software tests that repeat the implementation's incorrect assumption?
Check each test's expected values against the requirement rather than against the code. A test that shares the implementation's wrong assumption usually computes its expected result the same way the code does, was written alongside the code from the same reading of the requirement, and is later edited in the very commit that fixes the bug. In a repository's history, that last signal is the strongest: the fix changes both the code and the test's assertion.
The reliable proof is a requirement grounded test, written from the requirement text alone, that fails on the buggy revision and passes on the fixed one while the original test passed on both sides of the bug. A software test false confidence dataset is a collection of such cases, each linked to the requirement version, the code, the tests, the review and what happened in production.
Why coverage and mutation scores stay green when the assumption is shared
Coverage shows which lines ran; it does not show whether the expected values were right. The StrykerJS documentation makes the point directly: code coverage does not tell you everything about the effectiveness of tests, and mutation testing goes further by inserting small faults and checking whether any test fails. A mutant is killed when a test fails and survives when all tests pass.
Mutation testing still cannot see a shared wrong assumption. If the code and the test both encode the same misreading, changing the code breaks the test, so the mutant dies and the suite looks strong. What the suite checks is consistency with the implementation, not agreement with the requirement.
This is a form of what the research literature calls the test oracle problem. Barr, Harman, McMinn, Shahbaz and Yoo's survey in IEEE Transactions on Software Engineering describes it as the challenge of distinguishing correct from potentially incorrect behavior for a given input. Mutation tools remain worth running; Google's engineers report using diff-based mutation testing in code review, with mutants limited to covered and relevant lines. They answer a different question from the one this dataset asks.
The practical consequence is that no suite-level metric will rank these cases for you. A module with full coverage and a high mutation score can still hold a shared misreading, and a module with weak scores may hold none. Finding them takes evidence from outside the code: the requirement, the discussion around the fix and what users experienced. That is also why the cases are scarce in public data and worth collecting carefully when a company has them.
Signals in repository history that a test agreed with the bug
Mine history for the patterns below, then confirm each candidate by hand against the requirement. The signals differ in strength, and two of them produce many false positives if used alone.
A practical triage order is to start from fixing commits, because they are already linked to a known defect. Filter to fixes whose diff touches both production code and an assertion in a test file, then read the linked issue. Only after that pass is it worth scanning tests for expected values computed by project code, which is slower and noisier. Expect most candidates to drop out; a short list of confirmed cases is more useful than a long list of plausible ones.
| Signal | Where to find it | Strength | Common false positive |
|---|---|---|---|
| Fix commit changes an assertion's expected value | Diff of the fixing commit or pull request | Strong | Requirement changed between the original test and the fix |
| Expected value computed by calling the code under test | Test source, helper functions | Strong | Round-trip tests that intentionally compare two code paths |
| Snapshot or golden file regenerated in the fix | Snapshot files, golden outputs | Medium | Unrelated formatting change in the same regeneration |
| Test and code added in one commit with no requirement reference | Commit metadata, pull request description | Weak alone | Normal test-first or same-commit practice |
| Issue or review comment citing the requirement against the test | Issue tracker, review threads | Strong | Comment disputes the requirement rather than the test |
| Production incident or ticket on code with passing tests | Incident records, support tickets linked to commits | Medium | Failure caused by configuration, data or a dependency |
Worked case: a proration rule read the same wrong way twice
This case is illustrative and does not come from a real repository. The starting state: a billing service, a requirement document at version 3 stating that a mid-cycle plan upgrade is prorated over the remaining days of the cycle including the upgrade day, and a pull request implementing it.
The case is laid out in the order a reviewer would check it: the state of the code and tests before anyone noticed a problem, the records that link the evidence, the decision taken, and how the outcome was verified.
- Starting state: the developer counted remaining days from the day after the upgrade. The unit test built its expected value with the same day-count helper the code used, so it agreed with the code by construction. Line coverage on the module was complete and every mutant in the function was killed.
- Linked evidence: requirement version 3, section on upgrades; the original pull request, which cited the requirement by title but not by section; the test file; and the code review, which approved with one comment about naming.
- What surfaced it: over 7 weeks, support logged 23 tickets about upgrade charges that looked one day short. An engineer linked them to the billing module.
- Decision: the fixing pull request changed the day-count logic and the test's expected values in the same diff, and added a new test whose expected values were written out by hand from the requirement's worked example. The reviewer's approval cited the requirement section.
- Verified outcome: the new test fails on the buggy revision and passes on the fixed revision; the original test passes on the buggy revision and fails on the fixed one. After deployment, the reconciliation report showed no upgrade invoices off by a day across the next 2 cycles, and no further tickets of that type were linked.
The requirement grounded test failure example collection brief
Record each case with the fields below. The brief separates what the original test encoded from what the requirement says, and keeps every claim tied to a record a reviewer can open.
Checklist · 3 items
- Derive grounded expected values from the requirement's own examples or by hand calculation, never by calling project code.
- Where the requirement states a rule over many inputs, a property-based test is a good grounded test: the Hypothesis documentation describes writing tests that should pass for all inputs in a described range and letting the library choose inputs, including edge cases.
- Keep the original test unchanged in the case record even after it is fixed; it is the evidence of false confidence.
0 of 3 ticked. Tick each item as you go.
| Field | What it holds | Format | Acceptance rule |
|---|---|---|---|
| requirement_ref | Requirement document, section and version in force | Identifier, section, version, date | Version must predate the original test commit |
| shared_assumption | The misreading both code and test encoded | One plain sentence | Must contradict a quoted requirement passage |
| buggy_revision and fixed_revision | Commits before and after the fix | Commit hashes | Fix must be a single logical change or a minimized diff |
| original_test | Test that passed on the buggy revision | Path, test name, assertion lines | Must pass on buggy revision in the rebuilt environment |
| grounded_test | Test written from the requirement alone | Path, test name, how expected values were derived | Must fail on buggy and pass on fixed revision |
| environment | What a reviewer needs to rerun both revisions | Container image or build recipe, lockfile, run command | Two clean reruns give the same results |
| review_rationale | Why the fix was accepted | Review comment references | At least one comment ties the change to the requirement |
| production_evidence | What showed the bug and the result of the fix | Ticket, incident or report references with counts and window | Window stated; no claim beyond it |
| case_label | Type of case | Shared assumption, missing test, requirement changed, flaky, undetermined | Only shared assumption cases enter the core set |
| provenance | Whether the case is real or illustrative and its rights status | Real or illustrative; rights cleared yes or no | Illustrative cases are never mixed into evaluation splits |
Wrong assumption or changed requirement? The counterexample to rule out
Before labeling a case as a shared wrong assumption, confirm the requirement did not change. If version 3 said one thing when the test was written and version 4 said another when the fix landed, the original test was right at the time. Label that case requirement changed; it teaches change management, not oracle failure.
Checking this means keeping requirement documents under version control or at least with dated revisions. Many teams hold requirements in a wiki, a ticket description or a product brief that is edited in place. Where no history exists, look for the requirement as quoted at the time, for example in the original pull request, a design review or an acceptance test plan, and record that quotation as the requirement reference with its date.
Two more cases need to be ruled out. A missing test, where no test covered the behavior at all, is a coverage gap rather than false confidence. A flaky test, whose result depends on timing or execution order, says nothing reliable about either revision. Defects4J's inclusion criteria follow the same logic for bugs: a triggering test must fail before the fix and pass after it, and the failure must not be random or dependent on test execution order.
Rebuilding both revisions: the four results a second reviewer must see
A second reviewer can verify a case only if it ships with the environment and both revisions, and the reviewer can rerun the original and grounded tests and see the expected four results. Without that rerun, the case is a story about a bug, not verified evidence.
Public benchmarks show the minimum shape. The SWE-bench dataset guide lists, for each task, the repository, a base commit, the problem statement, the gold patch, a test patch and two test lists, FAIL_TO_PASS and PASS_TO_PASS. Defects4J labels buggy and fixed revisions and records the test methods that trigger each bug. A false confidence case adds one element neither needs: the original test that passed on the buggy revision, kept as recorded.
- Original test on buggy revision: passes.
- Original test on fixed revision: fails or was edited in the fix.
- Grounded test on buggy revision: fails.
- Grounded test on fixed revision: passes.
- All other tests: same result on both revisions, apart from known flaky tests listed separately.
Proof of the fix decision and of the production result
The decision is proven by a review record that ties the change to the requirement, and the outcome by evidence from after deployment, bounded by a stated window. A merged status or a closed issue shows that work stopped, not that the result was right.
Useful decision records include a reviewer comment citing the requirement section, a design note or an issue thread where the requirement was clarified. Useful outcome records include deployment references, reconciliation or monitoring reports, and the drop in linked tickets over a stated period. When only the merge exists, label the outcome undetermined rather than inferring success from silence.
Keep decision and outcome evidence separate in the record. A reviewer who approved the fix for the right reason can still have approved a fix that did not hold, and a fix that held can have been approved for reasons unrelated to the requirement. Buyers training models to review code or judge test quality need to see both, because each teaches a different judgment.
Rights questions in code, tests and requirement documents
Each case combines several kinds of material with different owners. Code and tests may belong to the company or to clients it built software for; requirement documents are often client deliverables under confidentiality terms; tickets that surfaced the bug carry customer data. Having a repository on your servers does not settle who may license it.
Confirm ownership of the repository, including contractor contributions, and check client contracts for restrictions on reuse of specifications. Scan test fixtures for production data before anything leaves the company, and replace customer details in linked tickets with placeholders while keeping counts and dates.
Checklist · 5 items
- Company-built product code and tests: usually company-owned, subject to contractor assignment terms and open source license obligations in dependencies.
- Code built for clients: often assigned to the client; the services agreement decides whether any reuse is possible.
- Requirement documents and specifications: may contain client confidential information or third-party standards text; quote only what the rights review clears.
- Support tickets and incident records: contain customer personal data; keep references, counts and dates, and remove identities.
- Review comments: written by employees in the course of work, but may name individuals; replace names with roles.
0 of 5 ticked. Tick each item as you go.
Turning false-confidence cases into a licensable collection
Cases where tests passed and the software was still wrong are hard to find in public code, because the evidence sits in private requirements, reviews and support tickets. A software company with linked history can assemble them in a way few public sources can.
Software companies can license such cases through SourceX, which manages the rights review over code, specifications and tickets, plus contracting, preparation and delivery. A brief eligibility assessment comes first and needs no repository access, and no case is delivered until the company has approved its scope and signed. Filling in the collection brief for a handful of fixed bugs is a quick way to see whether the history holds such cases.
Frequently asked questions
What is a software test false confidence dataset?
It is a set of cases where tests passed although the code was wrong, because the tests encoded the same misreading of the requirement. Each case links the requirement version, the buggy and fixed revisions, the original and requirement grounded tests, the review and production evidence.
Does mutation testing catch tests that share the code's wrong assumption?
Usually not. Mutation testing checks whether tests fail when the code changes, and a test built on the same misreading still fails when the code is mutated. It measures sensitivity to code changes, not agreement with the requirement.
How do you write a requirement grounded test?
Derive expected values from the requirement text or its worked examples, by hand or with an independent calculation, without calling the code under test. For rules over many inputs, a property-based test expresses the requirement as a property that should hold for all inputs in a range.
Can open source bugs be used for this kind of collection?
Some can, where an issue cites a specification and the fix changed a test's expected value. Most open source projects lack versioned requirements and production evidence, which is why private company history tends to yield stronger cases.
What makes a case unusable?
A requirement that changed between the test and the fix, a flaky original test, an environment no one can rebuild, or an outcome inferred only from a merged pull request. Label these separately rather than forcing them into the core set.
Sources
- Code coverage does not tell everything about test effectiveness; a mutant is killed when a test fails and survives when tests pass. Source
- The test oracle problem is the challenge of distinguishing correct from potentially incorrect behavior for a given input (Barr et al., IEEE TSE 2015). Source
- Google describes diff-based mutation testing in code review that omits lines without statement coverage and uninteresting lines. Source
- Defects4J bugs have buggy and fixed revisions and a triggering test that fails before the fix and passes after, not random or order dependent. Source
- Each SWE-bench instance lists repository, base commit, problem statement, gold patch, test patch, FAIL_TO_PASS and PASS_TO_PASS tests. Source
- Hypothesis tests should pass for all inputs in a described range, with the library choosing inputs including edge cases. Source
See if your company qualifies
A short company assessment. No data uploads are needed.