Skip to content

AI uses for records

Production bug evidence packet: linking issue, patch and outcome

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A production bug evidence packet links five things: the production signal that exposed the bug, the issue stating expected behavior, a test that fails before the patch and passes after, the reviewed commit and the deployment that shipped it, and a measured production outcome over a stated window. Closed issues and resolved statuses are workflow events, not proof; the packet treats them as timeline entries and verifies the outcome separately.

Checklist

Production bug evidence

This page covers production bug evidence packet: linking issue, patch and outcome.Illustrative

Key takeaways

  • A closed issue or resolved error shows a workflow rule fired, not that the bug is gone.
  • Pin the starting state with a base commit, dependency set and runtime so a reviewer can rebuild it.
  • Separate tests that prove the fix from tests that guard against collateral damage.
  • Code review proves the decision; suspect commits and timing only suggest it.
  • Verify the outcome with production data over a window chosen in advance, and export deployment history before it expires.
  • Clear code rights, production evidence and personal data separately before a packet leaves the company.

What evidence connects a production bug, its patch and the resulting outcome?

Five linked records do: the production signal that showed the bug (an error group, alert or customer ticket), the issue that stated expected behavior, a test that fails before the patch and passes after it, the reviewed commit and the deployment that carried it, and a production measurement afterward showing the signal stopped. Without the last two, a production bug evidence packet proves only that someone wrote code.

The packet is the unit a software company should keep for each bug it may later want to study, audit or offer as AI training or evaluation material. It starts from the state of the system before the fix, links every claim to a record in the tool that produced it, and ends with an outcome that was measured in production rather than inferred from a status field.

Two things are deliberately separate: the task outcome (did the patched software behave correctly) and the workflow outcome (was the issue closed, the ticket resolved, the release shipped). Status fields are easy to collect and often wrong about the first.

Why a closed issue and a merged patch prove less than they appear to

A closed issue records that a workflow rule fired, not that the bug is gone. On GitHub, closing keywords such as fixes or resolves in a pull request description close the linked issue when the pull request is merged into the default branch, and they are ignored for pull requests targeting any other branch. A keyword in a commit message also closes the issue on merge, but GitHub notes that the pull request containing that commit is not then listed as a linked pull request. Closure therefore depends on branch and wording, and the link you need for the packet can be missing even when the issue closed.

Error trackers have the same gap from the other side. Sentry's documentation explains that a plain Resolve treats any later event as a regression, while resolving in a specific release tells Sentry which versions count as expected noise; if the issue returns in a newer release than the one it was resolved in, its status changes to Regressed. A resolved status at export time is a snapshot, and the regression may arrive a week later.

So the packet treats closure and resolution as events in a timeline, each with a timestamp, and never as the outcome itself.

What the linked incident, issue, commit, test and outcome packet includes

The packet has eight parts, each with a stable identifier and a link back to its source system. The table is the asset; it lists what to capture, what each part proves and the check a reviewer runs before accepting it. Field names are suggestions.

Store each packet as an immutable snapshot: a manifest file listing every part with its source ID, retrieval time and content hash, plus the exported records themselves. Source systems keep changing after the fact. Issues get edited, comments deleted, branches rebased and error groups merged, so a packet that only holds links will drift from what the reviewer originally checked.

Give the packet its own version number and keep earlier versions. When a regression reopens the bug months later, the follow-up becomes a new version of the same packet rather than a new packet, which keeps the full history of one defect in one place.

What the linked incident, issue, commit, test and outcome packet includes
PartCaptureWhat it provesReviewer check
Production signalError group or alert ID, first and last seen, affected release, event count, sample event with personal data removedThe bug happened in production, in which versionFirst seen is on or after the affected release went live
Issue and requirementIssue ID, expected behavior as written, requirement or spec version, reproduction steps, link to the signalWhat correct behavior meant at the timeRequirement version predates the patch; signal link resolves
Starting stateBase commit SHA, dependency lockfile hash, runtime and OS versions, relevant feature flags and config keys (values redacted where sensitive)The exact code and environment the bug lived inA reviewer can check out the base commit and build it
Reproduction testTest ID, test file path, result on base commitThe bug is observable by a machine, not only a personTest fails on the base commit for the stated reason
Patch and reviewCommit SHAs, pull request ID, review comments, approver role, rationale and rejected alternativesWhat was changed, why, and who accepted itCommits are reachable from the merge; review is recorded before merge
Test resultsFail-to-pass tests, pass-to-pass tests, CI run ID and log linkThe patch fixes the bug without breaking checked behaviorEvery fail-to-pass test passes and no pass-to-pass test fails
Build and deploymentArtifact digest, build provenance, deployment ID, environment, status history with timestampsThe patched code is what actually reached productionArtifact provenance names the merge commit; deployment reached success
Production outcomeSignal state after deployment, observation window, regression flag, follow-up issuesThe bug stopped in production and stayed stopped for the windowNo new events in the window from versions at or after the fix

Can a separate reviewer rebuild the environment and verify the claimed engineering result?

Only if the packet pins the starting state precisely enough to rebuild it: a base commit, the dependency set that commit resolved to, and the runtime it ran on. If any of those is missing, the reproduction test may pass or fail for reasons unrelated to the bug.

Public coding benchmarks show the minimum. SWE-bench's dataset guide lists, for each task instance, the repository, the base commit, the problem statement, the gold patch, a separate test patch, and two test lists: FAIL_TO_PASS and PASS_TO_PASS. That structure separates the tests that demonstrate the fix from the tests that guard against collateral damage, and it is a good template for the test part of the packet even when the goal is internal audit rather than benchmarking.

Build provenance closes the gap between source and production. The SLSA build provenance specification describes provenance as verifiable information about where, when and how an artifact was produced, written so that consumers can verify the artifact was built as expected and others can rebuild it. If your pipeline already emits provenance, store it in the packet; if it does not, record the artifact digest and the CI run that produced it as the next best link.

Which linked record proves the decision, and which only suggests it?

The code review record proves the decision: the comments where reviewers asked for changes, the rationale the author gave and the approval that allowed the merge. Tool-generated links only suggest it, and the packet should label them that way.

Sentry's suspect commits feature is a good example of a useful hint. It shows the most recent commit to the code in a stack trace, with the author and the pull request, and it depends on a repository integration and correct code mappings. That narrows the search for the commit that introduced a bug, but the most recent change to a file is not necessarily the cause. Record it as a lead, and record the commit the team actually identified, with the evidence they used, as the finding.

Rejected alternatives deserve their own field. A reviewer who asked for a guard clause instead of a retry, or who blocked a fix that masked the error, is recording engineering judgment the final diff cannot show.

  • Proves: approved review, rationale in the pull request, failing test that the patch turns green.
  • Supports: suspect commit, blame history, timing correlation between a deploy and the first error.
  • Does not prove on its own: issue closed, ticket resolved, error status set to resolved, a plausible final message from the author.

Which record proves the outcome in production?

The outcome is proven by the production signal after deployment, compared against the deployed version, over a stated window. A deployment record ties the fix to a time and an environment; the error tracker or monitoring system then shows whether events from versions at or after the fix stopped.

At a glance

Which record proves the outcome in production

Capture deployment history early. GitHub's REST documentation for deployment statuses says previous statuses are retained for 90 days and then deleted, while the deployment's current status stays available. A packet assembled a year later can still see that a deployment succeeded, but not when it moved through each state, so export the status history while it exists or rely on your own pipeline logs.

Pick the observation window before looking at the data, and write it in the packet. A window chosen after the fact tends to end just before the regression.

  • Crash or exception bugs: no new events in the error group from fixed versions during the window.
  • Wrong-result bugs: a data check or reconciliation query that failed before the fix and passes after, run on production data.
  • Performance bugs: the latency or resource metric back within its target for the window, compared with the same period before the fix.
  • Customer-reported bugs: no reopened or new tickets citing the same symptom from accounts on fixed versions.
  • Intermittent bugs: a window long enough to include several occurrences of the triggering condition, stated as a count of trigger events rather than days alone.

A worked case: one bug traced from report to verified outcome

The case below is illustrative: a fictional field-service scheduling product, invented identifiers and invented counts and durations. No SourceX client supplied any of it. The middle column is a complete packet; the right column is a counterexample that looks finished but fails verification.

A worked case: one bug traced from report to verified outcome
StageComplete packet (illustrative)Counterexample (illustrative)Why the counterexample fails
Starting stateRelease 5.2 live; error group E-410 first seen 40 minutes after deploy: appointments saved with the wrong time zoneError group exported with no affected releaseCannot tie the bug to a version
Linked evidenceIssue 1182 links E-410 and states expected behavior per scheduling spec v3; base commit and lockfile hash recordedIssue text paraphrased from memory after the fixExpected behavior not recorded at the time
ReproductionNew test for daylight-saving boundary fails on base commitTest added in the same commit as the fix, never run on baseNo proof the test detects the bug
DecisionPull request 2207 reviewed; reviewer rejected storing local time, approved converting at the API edge; rationale in threadMerged with no review; suspect commit used as root causeDecision and cause are unsupported
DeploymentArtifact digest and provenance name the merge commit; release 5.2.1 deployment reached successIssue closed by keyword; no deployment recordClosure is not delivery
Verified outcomeNo E-410 events from 5.2.1 or later during the agreed 14-day windowE-410 marked resolved; events resume in 5.3 and the issue shows RegressedOutcome was a status, not a measurement

How to validate a packet before anyone relies on it

Validate packets with checks a script can run, and reject a packet on the first failure with the reason logged. A rejected packet with a clear reason is worth more than a repaired one whose links no longer match the source systems.

At a glance

How to validate a packet before anyone relies on it

Checklist · 7 items

  1. Links resolve: every ID points to a record that exists in its source system or in the packet archive.
  2. Order holds: affected release before first error, requirement version before patch, review before merge, merge before deployment, deployment before the observation window.
  3. Rebuild works: the base commit builds with the recorded dependency set, and the reproduction test fails on it.
  4. Tests agree: every fail-to-pass test passes on the merge commit and every pass-to-pass test still passes.
  5. Delivery is proven: the deployed artifact traces to the merge commit through provenance or a recorded CI run.
  6. Outcome is measured: the window is stated, data covers it, and any regression is recorded rather than trimmed.
  7. Packet is clean: secrets, customer data and personal identifiers are removed from events, logs and comments.

0 of 7 ticked. Tick each item as you go.

What must be cleared before a bug packet leaves the company

Clear three things separately: the right to the code, the content of the production evidence, and the people in the record. Read access to a repository says nothing about whether the company may license what is in it.

Code rights come first. Your own engineers' work usually belongs to the company, but repositories also hold vendored third-party code, open source under licenses with conditions, and sometimes code written for a client under a contract that assigns it to the client. Production evidence comes next: error events and support tickets often carry customer names, emails, request payloads and tokens, which must be removed or the packet left out. People come last: author names and emails in commits and review threads are personal data in many jurisdictions, so replace them with roles unless counsel confirms otherwise.

Postmortem action items are often the missing link here. PagerDuty's public postmortem template asks that each action item be filed as a ticket with consistent tags, which makes follow-up fixes easy to find and attach to the original packet; adopt the same habit so later fixes are not mistaken for new bugs.

When engineering history is worth assessing for licensing

A software company that can produce packets like this for a meaningful share of its production bugs holds unusually well-evidenced engineering history. For such a company, SourceX manages the clearance, contract, data preparation and delivery steps needed to license selected packets to AI developers as a dataset.

It starts with a brief company assessment, and no code or data needs to be uploaded to see whether the company qualifies. The company approves the scope and terms and signs before anything is delivered, so specific repositories, clients or years can be left out from the start.

Frequently asked questions

What is a production bug evidence packet?

It is a linked set of records for one bug: the production signal, the issue and requirement, the starting code and environment, a reproduction test, the reviewed patch, test results, the deployment, and a measured production outcome. Each part points to the system that produced it, so a reviewer can check every claim.

Is a merged pull request with a closing keyword enough evidence?

No. On GitHub, closing keywords only close an issue when the pull request merges into the default branch, and a closed issue does not show that the fix was deployed or worked. The packet needs a deployment record and a production measurement as well.

How long should the observation window be?

Long enough to cover the conditions that triggered the bug, such as a weekly batch job or a date boundary, and chosen before you look at the results. Write the window into the packet so anyone can check whether a regression fell inside it.

Are suspect commits from an error tracker proof of the cause?

They are a lead, not proof. Sentry describes suspect commits as the most recent commit to code in the stack trace, which narrows the search but can point to an unrelated change. Record the commit the team confirmed, with its evidence, as the finding.

Can we include bugs whose deployment history has expired?

Yes, but mark the delivery link as reconstructed from pipeline logs or release notes rather than from deployment records. GitHub, for example, deletes previous deployment statuses after 90 days, so export them promptly for future bugs.

Do we need the actual source code in the packet?

For rebuild and test verification, yes, at least the files touched and the commit context. If code rights are unclear, a packet with metadata, test names and review text can still show the decision and the outcome without the code.

Sources

  • GitHub explains that closing keywords link and close issues only for pull requests merged into the default branch, and that a keyword in a commit message closes the issue without listing the pull request as linked. Source
  • GitHub's REST documentation says previous deployment statuses are retained for 90 days, while a deployment's current status remains available. Source
  • SWE-bench's dataset guide lists each instance's repository, base commit, problem statement, gold patch, test patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
  • Sentry describes suspect commits as the most recent commit to the code in a stack trace, shown with its author and pull request, and requires a repository integration and code mappings. Source
  • Sentry explains that a plain Resolve treats later events as a regression, that issues can be resolved in a specific release, and that a recurrence in a newer release sets the status to Regressed. Source
  • The SLSA build provenance specification describes provenance as verifiable information about where, when and how an artifact was produced, so consumers can verify it and others can rebuild it. Source
  • PagerDuty's postmortem template asks that each action item be filed as a ticket carrying consistent tags. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies