AI uses for records
Code review disagreement dataset: why a working patch was rejected
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
When a reviewer rejects a patch that works, the record should keep the requirement version, the patch and its passing tests, the reviewer's authority, a rationale code with the specific evidence behind it, the alternative the reviewer proposed and what happened next, through deployment. A code review disagreement dataset built this way captures engineering judgment that a merged or rejected flag loses.
License terms
Code review disagreement
Key takeaways
- Review tools record that a reviewer objected and how strongly, rarely why in a structured form.
- Map each objection to one primary rationale code and link the comment or document that supports it.
- Record the reviewer's alternative and whether it was ever built, merged and deployed.
- Label a change as rejected only when an authorized reviewer's blocking objection stood.
- Pin base commit, dependencies and CI job so a second reviewer can confirm the patch really worked.
What evidence should explain why a reviewer rejected a functioning patch?
The record should hold the requirement the patch was written against, the patch and its passing tests, the reviewer's stated reason mapped to a rationale code, the specific evidence behind that reason, the alternative the reviewer proposed, what happened next, and the reviewer's authority to block. A rejection without those parts is just a red label on code that worked.
Working code gets rejected all the time for good reasons: it duplicates a component that already exists, solves the problem in the wrong layer, adds a dependency the team is trying to remove, or builds a feature the owners do not want. Those reasons are the most instructive part of a review history, and they are exactly what a pass or fail flag throws away.
This guide is for engineering leaders preparing review histories for internal analysis, evaluation sets or a licensing review. It gives an annotation guide for rationale and alternatives, a rationale code list, a worked example, the cases that should not be labeled as rejections, and the points where a rights reviewer has to take over. A code review disagreement dataset built this way captures judgment, not just outcomes.
Why do passing tests open the record instead of closing it?
Because review asks a different question from tests: not whether the code works, but whether the codebase is better with it. Google's published reviewer guidance makes that the senior principle, telling reviewers to favor approving a change once it definitely improves overall code health even if it is not perfect, while allowing them to deny approval for a feature they do not want in their system even when the code is well designed.
Research on review outcomes points the same way. Bacchelli and Bird's study of code review at Microsoft found that although finding defects is the main stated motivation, review also delivers knowledge transfer, team awareness and alternative solutions, and that understanding the code and the change is the central challenge. A later Google study by Sadowski and colleagues combined interviews, a survey and review logs for about nine million changes to describe why Google reviews code and how developers experience it.
If review produces alternatives and shared understanding, a dataset that keeps only verdicts loses most of what review is for. The useful unit is the disagreement: the patch, the objection, the evidence and the resolution.
What do review tools record about a rejection, and what do they leave out?
Review tools record that a reviewer objected and how strongly, but almost never why in a structured form. The reason lives in free-text comments, sometimes in a call or chat that never reached the review thread.
GitHub's documentation lists three review decisions: comment, approve and request changes, the last flagging feedback the author should address before merging. Gerrit's default Code-Review label is a scale: minus 2 means this shall not be submitted and blocks submission across all patch sets until the reviewer changes it, minus 1 means the reviewer would prefer it not be submitted as is but does not block, and plus 2 from someone with final say enables submission.
Some projects add a rule about reasons. The Apache Software Foundation's voting process lets a qualified minus 1 vote veto a code modification, but requires a technical justification showing why the change is bad, and states that a veto without one is invalid. That is the closest thing to a reason requirement in common practice, and it is a policy, not a field. Your annotation has to supply the structure.
| System signal | What it tells you | What it does not tell you | How to annotate |
|---|---|---|---|
| GitHub request changes | Reviewer wanted changes before merge | Whether the objection was blocking in practice, or why | Map to a rationale code from the comment text and linked discussion |
| Gerrit minus 2 | Submission blocked by this reviewer | The reason, or whether it was later withdrawn for a new patch set | Record block, reason code, and the patch set where it changed |
| Gerrit minus 1 | Preference against, not a block | Whether anyone acted on it | Record as non-blocking objection; do not count as rejection |
| Closed without merge | The change did not land | Whether review, the author, priorities or a duplicate ended it | Require a closure reason before labeling it a rejection |
| Apache-style veto | A formal block with a stated technical reason | Whether the justification was accepted as valid | Keep the justification and any later withdrawal |
The review rationale and alternative solution annotation guide
Annotate each disagreement as one record that joins the requirement, the patch, the review and the outcome. Fields marked required must be filled from evidence in the repository, tracker or review system; if the evidence is not there, record the field as unknown rather than inferring it. Example values are illustrative.
At a glance
The review rationale and alternative solution annotation guide
| Field | Required content | Illustrative value | Where the evidence lives |
|---|---|---|---|
| Requirement reference | Issue or spec ID and the version the author worked from | Issue 812, spec revision of record | Tracker history |
| Patch reference | Change or pull request ID, base commit and head commit | Change 4471, patch set 3 | Review system and repository |
| Functional evidence | Tests run, environment and result at the head commit | Unit and integration suites passed in CI job 9921 | CI logs and test reports |
| Reviewer role and authority | Whether the reviewer could block or approve the change | Code owner for the storage module | Code owners file or label permissions |
| Decision and strength | Blocking, non-blocking, or nit, as recorded | Blocking, request changes | Review event |
| Rationale code | One primary code and up to two secondary codes from the code list | Duplicates existing component; maintainability | Annotator, from comment text |
| Rationale evidence | The specific comment, document or measurement that supports the code | Comment links to existing cache module and its docs | Review thread |
| Alternative proposed | What the reviewer suggested instead, if anything, in one or two sentences | Extend the existing cache with a per-tenant key | Review thread |
| Resolution | Revised and merged, alternative merged separately, abandoned, escalated, or overridden | Alternative merged in change 4502 | Review system and repository |
| Outcome and confidence | Deployment and production signal tied to the merged result, plus annotator confidence (high, medium or low) with a note when the reason was inferred | Released in version of record, error rate unchanged over a stated window; confidence high | Deployment log, monitoring and annotator |
Which reasons can justify rejecting code that works?
Use a short, fixed list of rationale codes so annotators agree and later analysis can count them. Each code needs evidence that would let someone else confirm it, and a known mislabel to watch for.
Google's reviewer principles are a useful calibration for the list: technical facts and data overrule opinions and personal preferences, the style guide is the authority on pure style, and software design is almost never just a matter of preference. A rejection grounded only in taste should be coded as preference and treated as weak evidence.
| Rationale code | What it means | Evidence that confirms it | Common mislabel |
|---|---|---|---|
| Design or layer fit | The change solves the problem in the wrong place or against the architecture | Comment naming the intended layer or design document | Coded as style when it is structural |
| Duplicates existing component | Equivalent functionality already exists | Link to the existing module or prior change | Coded as design when the duplicate is concrete |
| Product or scope direction | Owners do not want the feature or behavior | Owner statement or roadmap reference | Coded as a code quality problem |
| Maintainability or complexity | Works now but adds complexity the team must carry | Specific construct and the simpler option | Used for any vague dislike |
| Performance or resource risk | Expected cost under realistic load | Benchmark, profile or stated limit | Claimed without a measurement |
| Security or data handling | Introduces exposure even if tests pass | Named risk and affected path, at a general level | Mixed up with test adequacy |
| Test adequacy | Passing tests do not cover the risky behavior | Missing case named by the reviewer | Coded as a bug when none was shown |
| Consistency or preference | Differs from codebase convention or reviewer taste | Reference to the convention, or none | Treated as strong evidence |
How should the reviewer's alternative be captured?
Record the alternative as the reviewer stated it, then link to whatever actually happened: a revised patch set, a separate change that implemented the suggestion, or nothing. An alternative that was never tried is still useful, but it must be labeled as proposed, not as validated.
Watch for deferred cleanup. Google's guidance on pushback notes that when an author promises to clean something up in a later change, experience shows the cleanup usually does not happen unless it follows immediately, so reviewers are advised to insist on it now unless there is an emergency. In an annotated history, an approval that relied on a promised follow-up should link the follow-up or record that none was found.
Severity labels in comments help separate real objections from suggestions. Google's comment guidance recommends prefixes such as Nit, Optional and FYI so authors know which comments are not mandatory. Where a team used those labels, map them directly; where it did not, annotators should mark severity as inferred.
Can a separate reviewer rebuild the environment and verify the claimed engineering result?
Only if the record pins what the patch ran against: the base commit, dependency lock files, build configuration, the test command and the CI job that produced the passing result. Without them, the claim that the patch worked is a status, not evidence.
Verification has two halves. The first is functional: check out the head commit, rebuild with the recorded dependencies and rerun the tests. The second is the reason: a second annotator reads the thread and assigns a rationale code without seeing the first annotator's choice. Disagreements between annotators are worth keeping, because they show which rationales are genuinely ambiguous.
Production evidence is a separate claim. If the alternative was merged and deployed, link the deployment record and say what was measured and for how long. If nobody measured, say that; do not let a merged alternative stand in for a better outcome.
Worked example (illustrative): a caching patch that duplicated an existing layer
This example is invented to show the annotation; it describes no real company or repository. An engineer adds an in-memory cache to a pricing lookup service to reduce latency for a large customer. Unit and integration tests pass and a local benchmark shows faster responses.
The code owner for the storage module blocks the change. The comment explains that a shared cache layer already exists, that a second cache would create two invalidation paths, and that the same result could come from adding a per-customer key to the existing layer. The reviewer links the existing module and its design note.
Annotation: decision blocking; primary code duplicates existing component, secondary code maintainability; evidence is the linked module and design note; alternative is extending the shared cache with a per-customer key. The author closes the original change, and a second change implementing the alternative merges a week later and ships in the next release. Outcome evidence is the deployment record plus the monitoring check the team used to confirm latency, cited as a window rather than as a headline number.
Confidence is high, because the reason, the evidence and the alternative are all in the thread. Had the owner and author agreed the design in a call that nobody wrote up, the same record would carry a low confidence and an inferred rationale, which is why Google's guidance asks reviewers to record the outcome of any offline discussion as a comment on the change.
Which closed changes should not be labeled as rejections?
Label a change as rejected only when a reviewer with authority blocked it for a stated reason and the block stood. Many unmerged changes fail that test, and labeling them as rejections teaches a model that working code is often wrong for reasons nobody gave.
Stop and bring in a rights reviewer before anything leaves the engineering team when the repository contains third-party or customer code, code under a license that restricts redistribution, or contributions from contractors whose agreements do not assign rights. Review threads also contain personal remarks about colleagues; treat names and comments about people under your privacy review, and remember that a rejection rationale describes code, not the competence of its author.
- Abandoned by the author with no reviewer objection: label as abandoned.
- Superseded by another change or a duplicate: link the winner and label as superseded.
- Only non-blocking objections or nits: label as unresolved suggestions, not rejections.
- Blocked and later overridden by a lead or maintainer: keep both the block and the override with reasons.
- Blocked because of release timing or a freeze: label as deferred.
- Reason given only in a call or chat with no written trace: keep with low confidence or exclude.
If annotated review histories are being considered for licensing
Review histories with recorded rationale, alternatives and outcomes are a strong source of judgment data for coding assistants and code review models, but the repository rights question comes first. A company can license only what it owns or controls, and contributor agreements decide much of that.
SourceX can take engineering records like these through licensing as an AI dataset, covering the rights review, contracting, preparation, delivery and payment. Delivery follows only after the scope and terms are approved and the agreement is signed, and a brief company assessment with nothing uploaded is the way to check whether engineering records are in scope.
Frequently asked questions
Why would a reviewer reject a patch that passes all its tests?
Tests show that code behaves as specified; review asks whether the codebase is better with it. Common reasons include duplicating an existing component, solving the problem in the wrong layer, adding complexity the team must maintain, or building a feature the owners do not want. Each is legitimate if the reviewer states it and points to evidence.
Does a GitHub request changes review count as a rejection?
Not by itself. It records that the reviewer wanted changes before merging, but the author may have revised the code and won approval, or the objection may never have been enforced. Label a rejection only when a blocking objection stood and the change did not merge for that reason.
What is the difference between a Gerrit minus 1 and minus 2?
In Gerrit's default Code-Review label, minus 1 means the reviewer would prefer the change not be submitted as is but does not block it. Minus 2 means it shall not be submitted, blocks submission and persists across patch sets until the reviewer changes it. Only a minus 2 that stood is a rejection in the sense used here.
How many rationale codes should a code review annotation scheme use?
Few enough that two annotators agree most of the time, typically around eight primary codes, with one primary and up to two secondary codes per record. Add a code only when annotators keep reaching for one that does not exist. A long list mostly measures annotator disagreement.
Should nit comments be included in a code review disagreement dataset?
Keep them as suggestions with their label, but do not count them as reasons for rejection. Nits are optional polish by definition, and Google's comment guidance recommends labeling them so authors know they are not mandatory. Mixing them with blocking objections inflates the apparent rejection rate.
Sources
- Google's code review guidance says reviewers should favor approving a change once it definitely improves overall code health even if imperfect, may deny approval for an unwanted feature even if well designed, holds that technical facts overrule opinions, and asks that offline discussion results be recorded on the change. Source
- Google's guidance recommends labeling comment severity with prefixes such as Nit, Optional and FYI so authors know which comments are not mandatory. Source
- Google's pushback guidance says promised later cleanups usually do not happen unless done immediately, so reviewers should usually insist on cleanup now except in an emergency. Source
- Gerrit's Code-Review label defines minus 2 as blocking submission across patch sets, minus 1 as a non-blocking preference against, and plus 2 as approval enabling submission. Source
- GitHub documents three review decisions: comment, approve and request changes. Source
- Apache's voting process lets a qualified minus 1 veto a code modification but requires a technical justification, and a veto without one is invalid. Source
- Bacchelli and Bird's study of code review at Microsoft found that finding defects is the main motivation but review also yields knowledge transfer, team awareness and alternative solutions, and that understanding is the central challenge. Source
- Sadowski and colleagues studied code review at Google through 12 interviews, a survey of 44 respondents and review logs for 9 million reviewed changes. Source
See if your company qualifies
A short company assessment. No data uploads are needed.