Skip to content

AI uses for records

Code review disagreement dataset: why a working patch was rejected

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

When a reviewer rejects a patch that works, the record should keep the requirement version, the patch and its passing tests, the reviewer's authority, a rationale code with the specific evidence behind it, the alternative the reviewer proposed and what happened next, through deployment. A code review disagreement dataset built this way captures engineering judgment that a merged or rejected flag loses.

License terms

Code review disagreement

This page covers code review disagreement dataset: why a working patch was rejected.Illustrative

Key takeaways

  • Review tools record that a reviewer objected and how strongly, rarely why in a structured form.
  • Map each objection to one primary rationale code and link the comment or document that supports it.
  • Record the reviewer's alternative and whether it was ever built, merged and deployed.
  • Label a change as rejected only when an authorized reviewer's blocking objection stood.
  • Pin base commit, dependencies and CI job so a second reviewer can confirm the patch really worked.

What evidence should explain why a reviewer rejected a functioning patch?

The record should hold the requirement the patch was written against, the patch and its passing tests, the reviewer's stated reason mapped to a rationale code, the specific evidence behind that reason, the alternative the reviewer proposed, what happened next, and the reviewer's authority to block. A rejection without those parts is just a red label on code that worked.

Working code gets rejected all the time for good reasons: it duplicates a component that already exists, solves the problem in the wrong layer, adds a dependency the team is trying to remove, or builds a feature the owners do not want. Those reasons are the most instructive part of a review history, and they are exactly what a pass or fail flag throws away.

This guide is for engineering leaders preparing review histories for internal analysis, evaluation sets or a licensing review. It gives an annotation guide for rationale and alternatives, a rationale code list, a worked example, the cases that should not be labeled as rejections, and the points where a rights reviewer has to take over. A code review disagreement dataset built this way captures judgment, not just outcomes.

Why do passing tests open the record instead of closing it?

Because review asks a different question from tests: not whether the code works, but whether the codebase is better with it. Google's published reviewer guidance makes that the senior principle, telling reviewers to favor approving a change once it definitely improves overall code health even if it is not perfect, while allowing them to deny approval for a feature they do not want in their system even when the code is well designed.

Research on review outcomes points the same way. Bacchelli and Bird's study of code review at Microsoft found that although finding defects is the main stated motivation, review also delivers knowledge transfer, team awareness and alternative solutions, and that understanding the code and the change is the central challenge. A later Google study by Sadowski and colleagues combined interviews, a survey and review logs for about nine million changes to describe why Google reviews code and how developers experience it.

If review produces alternatives and shared understanding, a dataset that keeps only verdicts loses most of what review is for. The useful unit is the disagreement: the patch, the objection, the evidence and the resolution.

What do review tools record about a rejection, and what do they leave out?

Review tools record that a reviewer objected and how strongly, but almost never why in a structured form. The reason lives in free-text comments, sometimes in a call or chat that never reached the review thread.

GitHub's documentation lists three review decisions: comment, approve and request changes, the last flagging feedback the author should address before merging. Gerrit's default Code-Review label is a scale: minus 2 means this shall not be submitted and blocks submission across all patch sets until the reviewer changes it, minus 1 means the reviewer would prefer it not be submitted as is but does not block, and plus 2 from someone with final say enables submission.

Some projects add a rule about reasons. The Apache Software Foundation's voting process lets a qualified minus 1 vote veto a code modification, but requires a technical justification showing why the change is bad, and states that a veto without one is invalid. That is the closest thing to a reason requirement in common practice, and it is a policy, not a field. Your annotation has to supply the structure.

What do review tools record about a rejection, and what do they leave out?
System signalWhat it tells youWhat it does not tell youHow to annotate
GitHub request changesReviewer wanted changes before mergeWhether the objection was blocking in practice, or whyMap to a rationale code from the comment text and linked discussion
Gerrit minus 2Submission blocked by this reviewerThe reason, or whether it was later withdrawn for a new patch setRecord block, reason code, and the patch set where it changed
Gerrit minus 1Preference against, not a blockWhether anyone acted on itRecord as non-blocking objection; do not count as rejection
Closed without mergeThe change did not landWhether review, the author, priorities or a duplicate ended itRequire a closure reason before labeling it a rejection
Apache-style vetoA formal block with a stated technical reasonWhether the justification was accepted as validKeep the justification and any later withdrawal

The review rationale and alternative solution annotation guide

Annotate each disagreement as one record that joins the requirement, the patch, the review and the outcome. Fields marked required must be filled from evidence in the repository, tracker or review system; if the evidence is not there, record the field as unknown rather than inferring it. Example values are illustrative.

At a glance

The review rationale and alternative solution annotation guide

The review rationale and alternative solution annotation guide
FieldRequired contentIllustrative valueWhere the evidence lives
Requirement referenceIssue or spec ID and the version the author worked fromIssue 812, spec revision of recordTracker history
Patch referenceChange or pull request ID, base commit and head commitChange 4471, patch set 3Review system and repository
Functional evidenceTests run, environment and result at the head commitUnit and integration suites passed in CI job 9921CI logs and test reports
Reviewer role and authorityWhether the reviewer could block or approve the changeCode owner for the storage moduleCode owners file or label permissions
Decision and strengthBlocking, non-blocking, or nit, as recordedBlocking, request changesReview event
Rationale codeOne primary code and up to two secondary codes from the code listDuplicates existing component; maintainabilityAnnotator, from comment text
Rationale evidenceThe specific comment, document or measurement that supports the codeComment links to existing cache module and its docsReview thread
Alternative proposedWhat the reviewer suggested instead, if anything, in one or two sentencesExtend the existing cache with a per-tenant keyReview thread
ResolutionRevised and merged, alternative merged separately, abandoned, escalated, or overriddenAlternative merged in change 4502Review system and repository
Outcome and confidenceDeployment and production signal tied to the merged result, plus annotator confidence (high, medium or low) with a note when the reason was inferredReleased in version of record, error rate unchanged over a stated window; confidence highDeployment log, monitoring and annotator

Which reasons can justify rejecting code that works?

Use a short, fixed list of rationale codes so annotators agree and later analysis can count them. Each code needs evidence that would let someone else confirm it, and a known mislabel to watch for.

Google's reviewer principles are a useful calibration for the list: technical facts and data overrule opinions and personal preferences, the style guide is the authority on pure style, and software design is almost never just a matter of preference. A rejection grounded only in taste should be coded as preference and treated as weak evidence.

Which reasons can justify rejecting code that works?
Rationale codeWhat it meansEvidence that confirms itCommon mislabel
Design or layer fitThe change solves the problem in the wrong place or against the architectureComment naming the intended layer or design documentCoded as style when it is structural
Duplicates existing componentEquivalent functionality already existsLink to the existing module or prior changeCoded as design when the duplicate is concrete
Product or scope directionOwners do not want the feature or behaviorOwner statement or roadmap referenceCoded as a code quality problem
Maintainability or complexityWorks now but adds complexity the team must carrySpecific construct and the simpler optionUsed for any vague dislike
Performance or resource riskExpected cost under realistic loadBenchmark, profile or stated limitClaimed without a measurement
Security or data handlingIntroduces exposure even if tests passNamed risk and affected path, at a general levelMixed up with test adequacy
Test adequacyPassing tests do not cover the risky behaviorMissing case named by the reviewerCoded as a bug when none was shown
Consistency or preferenceDiffers from codebase convention or reviewer tasteReference to the convention, or noneTreated as strong evidence

How should the reviewer's alternative be captured?

Record the alternative as the reviewer stated it, then link to whatever actually happened: a revised patch set, a separate change that implemented the suggestion, or nothing. An alternative that was never tried is still useful, but it must be labeled as proposed, not as validated.

Watch for deferred cleanup. Google's guidance on pushback notes that when an author promises to clean something up in a later change, experience shows the cleanup usually does not happen unless it follows immediately, so reviewers are advised to insist on it now unless there is an emergency. In an annotated history, an approval that relied on a promised follow-up should link the follow-up or record that none was found.

Severity labels in comments help separate real objections from suggestions. Google's comment guidance recommends prefixes such as Nit, Optional and FYI so authors know which comments are not mandatory. Where a team used those labels, map them directly; where it did not, annotators should mark severity as inferred.

Can a separate reviewer rebuild the environment and verify the claimed engineering result?

Only if the record pins what the patch ran against: the base commit, dependency lock files, build configuration, the test command and the CI job that produced the passing result. Without them, the claim that the patch worked is a status, not evidence.

Verification has two halves. The first is functional: check out the head commit, rebuild with the recorded dependencies and rerun the tests. The second is the reason: a second annotator reads the thread and assigns a rationale code without seeing the first annotator's choice. Disagreements between annotators are worth keeping, because they show which rationales are genuinely ambiguous.

Production evidence is a separate claim. If the alternative was merged and deployed, link the deployment record and say what was measured and for how long. If nobody measured, say that; do not let a merged alternative stand in for a better outcome.

Worked example (illustrative): a caching patch that duplicated an existing layer

This example is invented to show the annotation; it describes no real company or repository. An engineer adds an in-memory cache to a pricing lookup service to reduce latency for a large customer. Unit and integration tests pass and a local benchmark shows faster responses.

The code owner for the storage module blocks the change. The comment explains that a shared cache layer already exists, that a second cache would create two invalidation paths, and that the same result could come from adding a per-customer key to the existing layer. The reviewer links the existing module and its design note.

Annotation: decision blocking; primary code duplicates existing component, secondary code maintainability; evidence is the linked module and design note; alternative is extending the shared cache with a per-customer key. The author closes the original change, and a second change implementing the alternative merges a week later and ships in the next release. Outcome evidence is the deployment record plus the monitoring check the team used to confirm latency, cited as a window rather than as a headline number.

Confidence is high, because the reason, the evidence and the alternative are all in the thread. Had the owner and author agreed the design in a call that nobody wrote up, the same record would carry a low confidence and an inferred rationale, which is why Google's guidance asks reviewers to record the outcome of any offline discussion as a comment on the change.

Which closed changes should not be labeled as rejections?

Label a change as rejected only when a reviewer with authority blocked it for a stated reason and the block stood. Many unmerged changes fail that test, and labeling them as rejections teaches a model that working code is often wrong for reasons nobody gave.

Stop and bring in a rights reviewer before anything leaves the engineering team when the repository contains third-party or customer code, code under a license that restricts redistribution, or contributions from contractors whose agreements do not assign rights. Review threads also contain personal remarks about colleagues; treat names and comments about people under your privacy review, and remember that a rejection rationale describes code, not the competence of its author.

  • Abandoned by the author with no reviewer objection: label as abandoned.
  • Superseded by another change or a duplicate: link the winner and label as superseded.
  • Only non-blocking objections or nits: label as unresolved suggestions, not rejections.
  • Blocked and later overridden by a lead or maintainer: keep both the block and the override with reasons.
  • Blocked because of release timing or a freeze: label as deferred.
  • Reason given only in a call or chat with no written trace: keep with low confidence or exclude.

If annotated review histories are being considered for licensing

Review histories with recorded rationale, alternatives and outcomes are a strong source of judgment data for coding assistants and code review models, but the repository rights question comes first. A company can license only what it owns or controls, and contributor agreements decide much of that.

SourceX can take engineering records like these through licensing as an AI dataset, covering the rights review, contracting, preparation, delivery and payment. Delivery follows only after the scope and terms are approved and the agreement is signed, and a brief company assessment with nothing uploaded is the way to check whether engineering records are in scope.

Frequently asked questions

Why would a reviewer reject a patch that passes all its tests?

Tests show that code behaves as specified; review asks whether the codebase is better with it. Common reasons include duplicating an existing component, solving the problem in the wrong layer, adding complexity the team must maintain, or building a feature the owners do not want. Each is legitimate if the reviewer states it and points to evidence.

Does a GitHub request changes review count as a rejection?

Not by itself. It records that the reviewer wanted changes before merging, but the author may have revised the code and won approval, or the objection may never have been enforced. Label a rejection only when a blocking objection stood and the change did not merge for that reason.

What is the difference between a Gerrit minus 1 and minus 2?

In Gerrit's default Code-Review label, minus 1 means the reviewer would prefer the change not be submitted as is but does not block it. Minus 2 means it shall not be submitted, blocks submission and persists across patch sets until the reviewer changes it. Only a minus 2 that stood is a rejection in the sense used here.

How many rationale codes should a code review annotation scheme use?

Few enough that two annotators agree most of the time, typically around eight primary codes, with one primary and up to two secondary codes per record. Add a code only when annotators keep reaching for one that does not exist. A long list mostly measures annotator disagreement.

Should nit comments be included in a code review disagreement dataset?

Keep them as suggestions with their label, but do not count them as reasons for rejection. Nits are optional polish by definition, and Google's comment guidance recommends labeling them so authors know they are not mandatory. Mixing them with blocking objections inflates the apparent rejection rate.

Sources

  • Google's code review guidance says reviewers should favor approving a change once it definitely improves overall code health even if imperfect, may deny approval for an unwanted feature even if well designed, holds that technical facts overrule opinions, and asks that offline discussion results be recorded on the change. Source
  • Google's guidance recommends labeling comment severity with prefixes such as Nit, Optional and FYI so authors know which comments are not mandatory. Source
  • Google's pushback guidance says promised later cleanups usually do not happen unless done immediately, so reviewers should usually insist on cleanup now except in an emergency. Source
  • Gerrit's Code-Review label defines minus 2 as blocking submission across patch sets, minus 1 as a non-blocking preference against, and plus 2 as approval enabling submission. Source
  • GitHub documents three review decisions: comment, approve and request changes. Source
  • Apache's voting process lets a qualified minus 1 veto a code modification but requires a technical justification, and a veto without one is invalid. Source
  • Bacchelli and Bird's study of code review at Microsoft found that finding defects is the main motivation but review also yields knowledge transfer, team awareness and alternative solutions, and that understanding is the central challenge. Source
  • Sadowski and colleagues studied code review at Google through 12 interviews, a survey of 44 respondents and review logs for 9 million reviewed changes. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies