Systems and records
Software incident root cause uncertainty: keep it in training data
By SourceX Editorial · Updated
10 sections
Short answer
Engineering training data should keep an uncertain root cause uncertain by recording each explanation as a hypothesis with a status (confirmed, supported, plausible, ruled out or untested) and an evidence grade tied to linked artifacts such as traces, commits, tests and deploy records. Only a cause that was reproduced, or directly observed with no surviving alternative, should become the incident's confirmed label.
Record timeline
Software incident root cause
Key takeaways
- A postmortem's root-cause field usually holds the most plausible explanation at the time, not a verified one.
- Grade each hypothesis by the best evidence linked to it: reproduced, observed, consistent with the timeline, asserted or contradicted.
- Keep ruled-out and untested hypotheses; they show how explanations were tested.
- A grade a separate reviewer cannot reopen from linked artifacts should be lowered until the links are fixed.
- Deterministic defects reproduced by a regression test deserve a confident label; causes stated as a person's mistake should be rewritten as conditions.
How should engineering training data preserve uncertain root causes?
Store each causal explanation as a separate hypothesis with a status and an evidence grade, link every grade to the artifacts that earn it, and never collapse the set into one confident root-cause string unless the evidence reached the top grade. An incident record that says root cause: connection pool exhaustion is a claim; a record that says this hypothesis was observed in traces at the time, was not reproduced, and a competing hypothesis about a config change was ruled out by a rollback test is data a model can learn calibrated reasoning from.
The rest of this guide gives software and platform teams an evidence strength and hypothesis annotation rubric for their own incident and postmortem history, the verification checklist a separate reviewer would use, an illustrative worked case, the exception where a single cause is the honest answer, and the incidents to narrow or keep out before the history is shared.
Why does the root cause field in a postmortem overstate certainty?
Because the field asks for one answer and the analysis rarely produced one. PagerDuty's public postmortem guide, drawing on Richard Cook's How Complex Systems Fail, argues that major failures in complex systems come from a combination of contributing factors rather than a single root cause, and that the goal of analysis is to understand those factors. Google's SRE book frames the postmortem's purpose the same way, as understanding all contributing root causes and putting preventive actions in place.
In practice the root-cause field is filled under time pressure and often holds whichever explanation coincided with recovery: the rollback that preceded the error rate dropping, or the config change nobody had reviewed. Coincidence with recovery is evidence, but weak evidence, since recovery can come from load falling, a cache warming or another fix landing at the same time.
Training data inherits that overstatement. A model trained on thousands of single-cause postmortems learns to name one cause quickly and confidently, which is exactly the behavior an on-call engineer does not want from an assistant during an ambiguous incident.
Small incidents make the problem worse, not better. PagerDuty's guide recommends a postmortem for every major incident even when it later turns out to be a false alarm or recovered without intervention, and those are the records where the cause is most often left as a guess. Include them, graded honestly.
The evidence strength and hypothesis annotation rubric
Annotate every candidate cause in the incident, including the ones that were dismissed, with one evidence grade from the table and one status from the list below. Grades describe the best evidence that exists in the record, not how confident the team felt. If two annotators disagree on a grade, keep the lower one and note the disagreement.
- Status confirmed: grade E4, or E3 with no live competing hypothesis.
- Status supported: grade E3 or E2, with competing hypotheses still open.
- Status plausible: grade E1 only; keep it, but never as the incident's label.
- Status ruled out: grade E0; keep it, because a dismissed hypothesis shows the search.
- Status untested: raised but never checked; record why, such as data aged out or the team moved on.
| Grade | Meaning | Typical evidence in the record | What a reviewer must be able to do |
|---|---|---|---|
| E4 reproduced | The failure was reproduced on demand and the fix removed it | A failing test or replay before the patch, passing after, in a pinned environment | Rerun the test or replay from the linked commit and environment and see both results |
| E3 observed | The mechanism was seen directly while the incident was happening | Traces, logs or metrics from the incident window that show the faulty path, not just the symptom | Open the linked telemetry and find the mechanism at the stated times |
| E2 consistent | The hypothesis fits the timeline but the mechanism was not seen | Deploy or change timestamps that line up with onset, or recovery after a rollback | Confirm the timeline and see that no direct observation is claimed |
| E1 asserted | An engineer judged it likely; no artifact supports it yet | Postmortem prose, chat messages, meeting notes | Find the person's role and the reasoning, and confirm nothing stronger is linked |
| E0 contradicted | Evidence points against the hypothesis | A test that passed with the suspect change present, or telemetry showing the component was healthy | Open the counter-evidence and confirm it covers the incident window |
How should the uncertainty be worded so a model can learn from it?
Use a fixed vocabulary stored in fields, and keep free-text hedges out of the label. Phrases such as probably, likely or we believe drift between authors and cannot be compared across thousands of records.
There is established precedent for doing this. The U.S. intelligence community's analytic standards in ICD 203 require products to express and explain the uncertainties behind major judgments and to distinguish underlying information from the analyst's assumptions and judgments. In security incident recording, the VERIS schema stores a confidence value (high, medium, low or none) as data and lets an incident be recorded as suspected rather than confirmed. Software incident records can borrow both habits: the evidence grade carries how much is known, the status carries what was concluded, and a separate field holds assumptions the conclusion depends on.
Keep the postmortem's narrative text as it was written. It is valuable for teaching how engineers reason, but annotate it rather than editing it, so the original wording stays traceable.
Can a separate reviewer rebuild the environment and verify the claimed result?
Only if the record links every artifact the claim depends on, at the version that existed at the time. An E4 or E3 grade that a reviewer cannot reopen should be downgraded until the links are repaired. The checklist below is what that reviewer needs.
At a glance
Can a separate reviewer rebuild the environment and verify the claimed result
- Requirement or expected behavior: the spec, ticket or service level objective in force at the time, with its version.
- Repository and rights: which repositories hold the relevant code, and confirmation that your company owns or may license them; client-owned and third-party code is a separate question.
- Environment: commit identifiers for every deployed service involved, dependency lock files, configuration and feature-flag state at onset.
- Patch: the commit or pull request that changed behavior, with review comments showing the reasoning.
- Tests: a test that fails before the patch and passes after, or a replay harness with its inputs; note if none exists.
- Deployment record: when the patch reached each environment, from the deploy tool rather than from memory.
- Production evidence: the metric or error signal that returned to normal, over a window long enough to rule out coincidence, with that window stated.
How do you grade an existing postmortem archive without reinvestigating every incident?
Grade from the artifacts the record already links, and do not try to solve old incidents again. The rubric is designed so that a reviewer can assign a grade by opening links, not by rerunning the investigation; if a stronger grade would need new work, the honest grade is the one the record supports today.
Start with a pilot of a few dozen incidents spread across services and years, because older records usually have more broken links and thinner evidence. Two annotators grade the same pilot set independently, compare results and tighten the grade definitions wherever they split. After the pilot, one annotator per incident is usually enough, with a second pass on every record graded E4 or E3, since those are the grades most likely to be overstated.
Record who graded each hypothesis, as a role, and when. Store the grade and status in fields next to the original postmortem rather than inside it, so later changes to the rubric can be applied without touching the source text. When an artifact link has died, for example because a log retention window has passed, note the missing link explicitly; a grade that silently falls from E3 to E1 hides why it fell.
Expect many incidents to end with no hypothesis above E2. That is not a failed annotation project. It is an accurate picture of how often teams close incidents on a reasonable but unverified explanation, which is exactly what the dataset is meant to show.
Worked example (illustrative): a checkout latency incident with three explanations
Everything here is invented. A software company's checkout API slowed for about forty minutes after a routine deploy. The postmortem's root-cause field says database connection pool exhaustion. Annotating the record with the rubric gives a more honest picture.
- The first record named the mechanism (pool exhaustion) as the cause and left out the trigger (retry logic); the rubric kept both.
- The confirmation came later and is recorded as a dated change, so a model can see what was known at each point.
- Without the follow-up replay, the honest final state would have been two supported hypotheses, and that is a valid training example.
| Hypothesis | Evidence linked | Grade and status | Effect on the label |
|---|---|---|---|
| Connection pool exhaustion in the checkout service | Traces from the incident window show requests waiting for connections; pool metrics at maximum | E3, supported | Kept as the leading explanation, not as confirmed |
| New retry logic in the deploy multiplied database calls | Deploy timestamp matches onset; rollback preceded recovery; no test reproduces the multiplication | E2, supported | Kept as a contributing factor candidate |
| Slow queries from a database index change | Query metrics show normal latency throughout the window | E0, ruled out | Kept, to show the search |
| Follow-up replay three weeks later | A load replay with the retry change reproduces pool exhaustion; without it, no exhaustion | Retry logic rises to E4, confirmed; pool exhaustion becomes the mechanism | Label updated with the replay date, original annotation kept |
When is a single confident cause the right label?
When the evidence reached E4 and no competing hypothesis survived. Deterministic defects are common: an off-by-one in a date calculation, reproduced by a regression test that fails before the fix and passes after, does not need artificial hedging. Adding doubt where the evidence is strong is miscalibration in the other direction.
One exception runs the other way. When the recorded cause is a person, such as engineer ran the wrong command, rewrite it as the conditions that made the action possible. PagerDuty's guide states that an individual's action should never be considered a root cause and suggests looking at factors such as a script without rate limiting or out-of-date documentation. Names should not appear in the annotation at all; use roles.
Which incident records should be narrowed or kept out?
Narrow or exclude anything whose evidence would expose people, customers or unfixed weaknesses, and anything whose code or data your company does not have the right to license. Incident history is attractive precisely because it is candid, which is also why it needs a careful pass.
- Security incidents and vulnerabilities that are not fully fixed and disclosed, including exploit details in traces or tickets.
- Customer data that appears in logs, payloads or screenshots attached to the incident.
- Code from client engagements, vendors or third-party repositories that your agreements do not let you share.
- Postmortems tied to litigation, regulatory inquiries or employment matters.
- Chat transcripts that identify individuals; keep the reasoning, replace names with roles.
- Secrets of any kind found in configuration snapshots or environment dumps.
Can annotated incident history be licensed for AI work?
Software companies with years of postmortems, deploy records and linked commits hold something public benchmarks rarely have: real failures with the evidence trail that shows how certain anyone could be. Through SourceX, a software company can license a chosen slice of that incident history to AI developers, with the rights review, contract, preparation and delivery run as one process.
The engineering organization keeps control of what goes out: it picks the postmortems, repositories and years, and it approves scope and terms and signs before any record leaves. Qualification starts with a brief assessment of the company itself, and no postmortem has to be uploaded to complete it.
How often does a published postmortem hedge the cause it names?
Rarely, and a hedge written into the timeline can vanish by the time the summary is written. In the UK Ministry of Justice Cloud Platform team's public incident archive (39 reports at commit 8a52e24, read on 2026-10-27), 25 reports contain at least one sentence, report titles included, with a cause word such as cause, due to, because, triggered or resulted in, 52 sentences in all. Only 5 of the 52 also carry a hedge word such as suspect, possible, likely or might, and 2 of those hedge something other than the cause: which secrets may have been exposed and which users may have been affected. That leaves 3 hedged cause statements in 39 reports.
The report on Prometheus restarts in November 2023 shows how a hedge gets lost. Its timeline entry says that, after investigating the logs, the team "found that one possible root cause might be the readiness Probe failure prior to the restart of prometheus" and raised the probe timeout. Its resolution section drops the hedge: "Team identified that the readiness probe was failing and the prometheus was restarted." Its review actions still list four open questions about what else might cause or feed the restarts, from data ingestion to backups.
Under the page's rubric that explanation is E2, consistent: recovery followed the timeout change, nothing reproduced the failure, and the open questions remain untested, so its status is supported, not confirmed. The mechanism needs its own evidence too. Kubernetes documentation says a failed readiness probe removes the Pod's IP address from the EndpointSlices of matching Services, while the kubelet restarts a container that fails its liveness probe and kills one that fails its startup probe, so a record that ties readiness failures to restarts has to show the link.
How does the evidence-grade rubric score statements from real postmortems?
It sorts them by what each record can show, and most land at E1 or E2. The six statements below come from five Ministry of Justice reports and were graded by one annotator with the page's grades and statuses. No second annotator has graded them yet, so they are worked annotation examples, not a reviewer-agreement result:
| Statement in the public report | Report | Grade and status | Why |
|---|---|---|---|
| Disk space and other resources, "the usual suspects, were ruled out as the cause" | Prometheus unreachable, February 2020 | Ruled out as reported; held at E1 until the check is linked | No counter-evidence is attached, so a reviewer cannot reopen the exclusion |
| "We suspect an intermittent & external networking issue to be the cause of this outage." | Same report, resolution | E1 asserted; plausible | A suspicion is the only cause offered; keep it, never as the incident's label |
| AWS's statement, quoted in the report, that the root cause was a latent software race condition triggered when some health checking instances were restarted | Slow ingress traffic, July 2022 | E1 in this record; plausible; asserted by the vendor | The vendor may hold stronger evidence, but the report holds only the quotation |
| Descheduler evictions after a replica increase caused the CPU spike, and a caching change "appears to have generated the spike" | Cluster CPU spike, February 2023 | Both E2; both supported; neither confirmed | Two causes for one spike are never reconciled, so a single label would pick one arbitrarily |
| The readiness probe timeout caused the restarts | Prometheus restarts, November 2023 | E2; supported; four open questions untested | Recovery followed the change; the resolution's confident wording adds no evidence |
| "Planned change reverted to exclude that as a root cause but that didnt help" | Prometheus restarts, April 2024 | E0 contradicted; ruled out | Removing the suspect left the failure in place, an intervention any reviewer can read in the timeline |
How should a dataset record a cause that is later withdrawn, revised or only guessed at?
As typed, dated events against the claim, so the history of a cause is data rather than an overwrite. The page's worked example covers a cause that a later replay confirmed; four other kinds of revision need rules of their own, followed here by the request terms they imply:
- Retraction: a cause that a report stated and later withdrew keeps its row with status ruled out, the date, and the artifact or intervention that withdrew it, so a model sees a confident claim that did not survive.
- Wording drift: when the timeline hedges and the summary does not, annotate both sentences, grade the claim once from its evidence and record the stronger wording as a flag, never as an upgrade.
- Look-alike correction: a later incident on the same component is not a correction unless someone links the two. The Ministry of Justice's April 2024 Prometheus report traces restarts to a startup probe too short for a long write-ahead log reload, and records the reason for the longer reload three days after repair, but it does not mention the November 2023 readiness-probe explanation; a dataset may join them only if a reviewer records the link and its evidence.
- Untestable speculation: motive and intent cannot be graded. The Rust project's report on an October 2018 crates.io incident confines its root-cause section to what happened, an account uploading packages fast enough to get the servers throttled by GitHub, and sets apart a sentence that begins "While it is impossible to tell a user's intent, many, including the team, have speculated"; such sentences belong in a speculation span with no grade.
- Request terms: a buyer asking for root-cause evidence should ask for the cause sentences as written, their grades and statuses, and the revision events with dates, so a test set can be cut as of any day.
Sources for this section
- In the Ministry of Justice Cloud Platform team's public incident archive (39 reports at commit 8a52e24), 25 reports contain 52 sentences with a cause word and 5 of those sentences also carry a hedge word, under the counting rules stated in the writer notes. Source
- The November 2023 Prometheus report's timeline says one possible root cause might be the readiness probe failure, its resolution says the team identified that the readiness probe was failing, and its review actions list further questions about other causes. Source
- Kubernetes documentation says a failed readiness probe leads the EndpointSlice controller to remove the Pod's IP address from matching Services, the kubelet restarts a container that fails its liveness probe beyond the configured tolerance, and kills a container that fails its startup probe. Source
- The February 2020 Prometheus report says disk space and other resources were ruled out as the cause and, in its resolution, that the team suspects an intermittent and external networking issue. Source
- The July 2022 slow ingress report quotes AWS saying the root cause was a latent software race condition triggered when some health checking instances were restarted. Source
- The February 2023 CPU spike report attributes pod restarts to the Descheduler's RemoveDuplicates strategy after a replica increase and also records a caching change that appears to have generated the spike. Source
- The April 2024 Prometheus report records reverting a planned change to exclude it as a root cause without effect, a startup probe failure during a long WAL reload, and the reason for the longer reload identified on 2024-04-18. Source
- The Rust project's update on the October 15, 2018 crates.io incident gives a root-cause section about an account whose upload rate got the servers throttled by GitHub, and separately notes that many, including the team, speculated about the user's intent. Source
Frequently asked questions
What is the difference between a root cause and a contributing factor?
A root cause is a single explanation offered as the reason an incident happened. A contributing factor is one of several conditions that together made it possible. Many incident practitioners, including PagerDuty's public postmortem guide, argue that complex system failures rarely have a single root cause, so recording contributing factors with their evidence is usually more accurate.
How do you record a root cause that was never confirmed?
Record it as a hypothesis with a status such as supported or plausible and an evidence grade that reflects the best artifact linked to it. Keep competing hypotheses and any that were ruled out. Do not copy it into a single confirmed root-cause field.
Should ruled-out hypotheses be kept in incident training data?
Yes. They show the search an engineer performed and the evidence that excluded each option. A model that only sees final answers never learns how explanations are tested and dropped.
What evidence confirms a software incident root cause?
The strongest evidence is reproduction: a test or replay that fails before the fix and passes after, in an environment pinned to the versions running at the time. Direct observation of the mechanism in traces or logs from the incident window is the next strongest. Timing that lines up with a deploy or rollback is consistent with a cause but does not confirm it.
Can postmortems that name the engineer involved be used as training data?
Replace names with roles and rewrite any cause stated as a person's mistake into the conditions that allowed it. The reasoning in the postmortem is useful; the attribution to an individual is neither needed nor appropriate.
Sources
- PagerDuty's postmortem guide, citing Richard Cook, says there is no single root cause of major failure in complex systems but a combination of contributing factors, and that an individual's action should never be considered a root cause. Source
- PagerDuty's guide advises against blaming human error and points to contributing factors such as a script without rate limiting or out-of-date documentation. Source
- PagerDuty's guide recommends a postmortem for every major incident, including those later found to be a false alarm or that recovered without intervention. Source
- Google's SRE book says a postmortem documents the incident, seeks to understand all contributing root causes and puts preventive actions in place, focusing on causes without indicting individuals. Source
- The VERIS incident schema stores a confidence value of High, Medium, Low or None and records an incident as Confirmed, Suspected, Near miss or False positive. Source
- ICD 203's analytic tradecraft standards require products to properly express and explain uncertainties in major judgments and to distinguish underlying information from analysts' assumptions and judgments. Source
See if your company qualifies
A short company assessment. No data uploads are needed.