Systems and records
IT incident postmortem linkage: tying each lesson to its evidence
By SourceX Editorial · Updated
10 sections
Short answer
Treat every lesson in a postmortem as a claim and annotate it with pointers to the records that support it: the alert, incident timeline entries, decisions in chat, the deploy or change record, the runbook version and the follow-up ticket. Grade each link by strength, flag what was only learned in hindsight, and redact secrets first. The result shows why a decision was made, not just that a ticket closed.
Record timeline
IT incident postmortem
Key takeaways
- Split a postmortem into claims and attach each one to the specific alert, timeline entry, message, deploy or ticket that supports it.
- Classify claims first: timeline facts, impact, steps, rationale, factors, hindsight findings and action items need different evidence.
- Grade every link as direct, corroborating, inferred or none, and keep contradicting evidence rather than dropping it.
- Record whether responders could have known each piece of evidence at the time, so hindsight is never passed off as reasoning.
- Rotate exposed secrets, redact with stable placeholders, and hold back security incidents until a security reviewer clears them.
How do you connect a postmortem lesson to the incident evidence behind it?
Split the postmortem into individual claims, then attach to each claim a pointer to the record that supports it: the alert, a timeline entry in the incident tool, a decision in the chat channel, a deploy or change record, the runbook version that was followed, or the ticket that tracked the fix. Each pointer gets a strength grade, and claims with no pointer are marked as unsupported rather than quietly kept.
This is a small extension of what good postmortem practice already asks for. Google's SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes and the follow-up actions. PagerDuty's public postmortem guide goes further on evidence: for each item in the timeline, identify a metric or third-party page where the data came from, such as a monitoring graph or a log search, so the timeline stays rooted in fact rather than opinion.
Linkage turns those habits into a structure a second reader can check. It is what makes an incident history useful for training or evaluating an AI system that reasons about operations, because the model can see why a responder acted, not only that a ticket closed.
Why a closed ticket and a final configuration do not explain the decision
A closed ticket records an outcome, and a final configuration records an end state; neither records the options a responder weighed, the signal that changed their mind, or what they did not yet know. That reasoning lives in the timeline, the alert history and the chat thread, and it disappears if the dataset keeps only the ticket.
Atlassian's incident handbook treats the postmortem as its own ticket linked to the incident ticket, with a timeline in UTC covering lead-up, impact start, detection, escalation and end of impact. That linking is the backbone. The annotation guide adds the missing layer: which line of the postmortem is supported by which entry on that backbone.
PagerDuty's guide adds a distinction worth keeping. It asks postmortem owners to include information the team did not know during the incident but wishes it had, found later in monitoring, logs and deployments. A lesson built on hindsight is legitimate, but it must not be presented as the responder's reasoning at the time. The annotation guide records that difference explicitly.
Which statements in a postmortem need evidence, and which cannot have it?
Not every sentence in a postmortem is the same kind of claim, and each kind has a different best possible evidence. Classifying claims first keeps annotators from hunting for proof that cannot exist.
| Claim type | Example (illustrative) | Evidence that can support it | What evidence cannot settle |
|---|---|---|---|
| Timeline fact | Error rate on the checkout API rose at 14:02 UTC | Monitoring query or alert record with timestamp | Why the rise began |
| Impact statement | About one in five checkout requests failed for 38 minutes | Request logs or SLO dashboard export for the window | Business impact beyond the measured metric |
| Detection and response step | On-call acknowledged the page and rolled back release 412 | Paging tool log entry, deploy tool record | Whether the rollback was the best option |
| Decision rationale | Rollback chosen over hotfix because the diff was large | Chat message or incident note written at the time | Rationale reconstructed afterward |
| Contributing factor | Connection pool limit was lower in the new config | Config diff, change ticket, reproduction in staging | That it was the only factor |
| Hindsight finding | A capacity warning fired two days earlier and was missed | Alert history found during the review | That responders could have known it at the time |
| Action item | Add a presubmit check for pool size changes | Follow-up ticket and the merged change that closed it | That the action prevents recurrence |
The claim and evidence annotation guide
Each annotation is one row that joins a claim to one piece of evidence. A claim with three supporting records has three rows. The structure borrows from the W3C PROV ontology, which models provenance with entities, the activities that generate them and the agents responsible: here the claim and the evidence are entities, the incident response and the review are activities, and responders and annotators are agents.
At a glance
The claim and evidence annotation guide
- Annotate from the earliest evidence forward, as PagerDuty advises for timelines, so the record of what was known does not get rewritten by what was learned later.
- Never merge two claims into one row; if a sentence makes two assertions, split it into two spans.
- A claim with only inferred or no evidence stays in the record with that grade; it is useful as a negative example and honest about the gap.
- Keep contradicting evidence. A postmortem line that the logs contradict is one of the most informative cases in the set.
| Field | What to record | Allowed values or format | Check before accepting |
|---|---|---|---|
| Claim ID and span | The postmortem ID plus the exact sentence or bullet being annotated | PM-ID.section.n with the quoted text | Span quoted verbatim, not paraphrased |
| Claim type | One type from the classification table | Timeline, impact, step, rationale, factor, hindsight, action | Type matches the evidence sought |
| Evidence pointer | System and record identifier of the supporting item | Tool name plus incident, alert, message, deploy, ticket or commit ID | Pointer resolves in the export |
| Evidence time | When the evidence was created | UTC timestamp | Within or before the incident window, or flagged |
| Known at the time | Whether responders could have seen it during the incident | Yes, no, unclear | Hindsight claims must be no or unclear |
| Relation | How the evidence bears on the claim | Supports, contradicts, context only | Contradicting evidence is kept, not dropped |
| Strength | How directly the evidence proves the claim | Direct, corroborating, inferred, none | Inferred needs a written reasoning note |
| Redaction status | What was removed from the evidence and claim text | Clean, redacted with rule IDs, withheld | Second reviewer confirmed for withheld items |
| Annotator and date | Who made the link and when | Name or role, date | Different person reviewed a sample |
How strong does a link need to be?
Grade by what a skeptical reader could verify without asking anyone. Direct evidence shows the claimed fact itself, such as an alert record with the stated timestamp. Corroborating evidence makes the claim likely without showing it, such as a chat message saying a rollback started plus a deploy record two minutes later. Inferred means the annotator reasoned from evidence to the claim and wrote down how. None means nothing in the export supports it.
Decision rationale deserves the strictest test. A sentence such as we chose to roll back because the hotfix was risky is direct only when a note or message written during the incident says so. If the only source is the postmortem meeting, grade it inferred and mark known at the time as unclear, because a reason given afterward can differ from the reason that drove the choice.
Grades make the set filterable. A team building a dataset for decision reasoning can keep only cases where every rationale claim is direct, while a set for timeline reconstruction can accept corroborating links.
Linking an action item to the change that actually shipped
Follow each action item to a ticket, and each ticket to the change that closed it, then check that the closing change matches the item's stated completion condition. PagerDuty's guide, citing Google authors writing in USENIX ;login:, recommends action items that are actionable, specific and bounded, meaning each states how to tell when it is finished. A bounded item gives the annotator a testable condition.
Tool behavior can hide real links. GitHub's documentation says closing keywords in a pull request description create a link only when the pull request targets the default branch, and that a closing keyword in a commit message closes the issue on merge without listing the pull request as linked. An action item can therefore be done with no visible link, or show a link to work on a branch that never merged. Search commit messages as well as the linked pull request field before grading an action item as unresolved.
Worked example (illustrative): an expired certificate at a payments API
This case is invented to show the method; identifiers, times and counts are illustrative. A software company's postmortem for a 47-minute outage contains 22 sentences. Annotators classify them as 9 timeline facts, 2 impact statements, 4 response steps, 2 decision rationales, 2 contributing factors, 1 hindsight finding and 2 action items.
The timeline facts link directly to alert and paging records. One impact statement, that most failed requests came from mobile clients, has only corroborating support from a partial log sample and is graded accordingly. One rationale, that the team renewed the certificate by hand rather than wait for automation, links directly to a chat message sent at the time. The other rationale was first written in the review meeting and is graded inferred, known at the time unclear.
The hindsight finding, a renewal warning email sent to a former employee's inbox nine days earlier, is graded direct for the fact and no for known at the time. One action item links to a merged change that moves renewal alerts to a team alias; the other has an open ticket and is recorded as unresolved. The certificate's private key path and the internal hostname appear in two chat messages, so both are redacted under named rules before the case is released.
What must come out before an annotated incident leaves engineering?
Credentials, session identifiers, connection strings, personal data and any detail an attacker could act on come out first, and anything that was ever exposed as a live secret is rotated before the record is shared. The OWASP Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than recorded, including session identification values, access tokens, authentication passwords, database connection strings, encryption keys, sensitive personal data and commercially sensitive information. Postmortem evidence pulled from logs and chat inherits all of those risks.
GitHub's guidance on removing sensitive data makes the order clear for secrets: revoke or rotate first, because a rotated secret can no longer be used even if a copy survives. For incident datasets, redaction is a second line, not the first.
Some detail is sensitive because it is operational rather than secret: internal hostnames, network ranges, unpatched version numbers, and the exact steps that triggered a failure. Replace them with stable placeholders, such as host-A and host-B, so links between records still resolve after redaction, and record each replacement rule in the redaction status field.
Which incidents to hold back from an evidence-linked set
Hold back any incident where the evidence itself is the risk. NIST SP 800-61 Revision 3 frames incident response lessons as continuous improvement that feeds every function of a cybersecurity program, which is the right use for security incidents: inside the organization, under its own controls.
- Security incidents with an open investigation, law enforcement involvement or unpatched exposure: exclude until a security reviewer clears them.
- Incidents where most rationale claims grade inferred or none: the case teaches little about decisions; keep it only for timeline tasks.
- Incidents that name individual blame in the source records: the blameless practice PagerDuty's guide describes makes these unrepresentative and sensitive; exclude or rewrite with a reviewer.
- Incidents involving a customer's environment or data: the customer agreement decides, so hold until it is read.
- Incidents whose redaction would remove the facts the lessons depend on: exclude rather than ship a record that no longer supports its own claims.
Turning evidence-linked incident histories into a licensable dataset
Incident histories where each lesson points to its evidence are among the more useful operations records an IT or software company holds. SourceX can help such a company license a selected slice of them to AI developers, running rights review, contracting, preparation and delivery; a short assessment with nothing uploaded comes first, and scope, terms and signature all precede delivery.
Frequently asked questions
What is postmortem linkage in IT incident management?
It is the practice of connecting each statement in a postmortem to the incident records that support it, such as alerts, timeline entries, chat decisions, deploy records and follow-up tickets. Linkage lets a second reader verify a lesson instead of trusting the write-up.
How is a hindsight finding different from a contributing factor?
A contributing factor is a condition that helped cause the incident. A hindsight finding is evidence the team only discovered afterward, such as a warning nobody saw. Both are valid, but a hindsight finding should never be presented as what responders knew while they were acting.
Do we need the original chat logs, or is the postmortem enough?
For decision reasoning, the original messages or incident notes matter, because a rationale written during the incident is stronger evidence than one written at the review meeting. The postmortem alone supports timeline and action item tasks better than decision tasks.
What if a postmortem claim has no supporting evidence?
Keep the claim, grade it none, and say so. Unsupported claims are informative for evaluating systems that must tell reasoning from assertion, and hiding them makes the dataset look stronger than it is.
Who should review an annotated incident before it is shared?
An IT operations lead who can judge the claim grades and a security reviewer who can confirm redaction and decide on held-back incidents. Customer or contract questions go to whoever owns the relevant agreement.
Sources
- Google's SRE book defines a postmortem as a written record of an incident, its impact, mitigation actions, root causes and follow-up actions. Source
- PagerDuty's postmortem guide asks for a metric or source page behind every timeline item, hindsight information found later, and actionable, specific, bounded action items. Source
- PagerDuty describes postmortems as blame-free analysis that produces a record of what went wrong and steps to prevent recurrence. Source
- Atlassian's incident handbook links the postmortem ticket to the incident ticket and keeps a UTC timeline from lead-up to end of impact. Source
- The W3C PROV ontology models provenance through entities, activities and agents, with agents ascribed responsibility for activities or entities. Source
- GitHub closing keywords link a pull request to an issue only on the default branch, and commit message keywords close issues without listing the pull request as linked. Source
- OWASP lists session identifiers, access tokens, passwords, connection strings, keys and sensitive personal data among data to remove or mask rather than log. Source
- GitHub advises revoking or rotating a leaked secret as the first step before rewriting any history. Source
- NIST SP 800-61 Revision 3 feeds lessons learned from incident response into improvement that informs all cybersecurity functions. Source
See if your company qualifies
A short company assessment. No data uploads are needed.