Skip to content

Privacy and preparation

IT incident timeline dataset: what to keep beyond the closed ticket

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

An IT incident training case should preserve the timeline, not just the closed ticket: the alerts that fired, what responders believed at each moment, the actions they took and who approved them, the runbook version followed, and measured evidence that service recovered. Each record needs a UTC timestamp and a link to its source, with secrets and sensitive operational detail removed before the case leaves the operations team.

Record timeline

IT incident timeline dataset

This page covers IT incident timeline dataset: what to keep beyond the closed ticket.Illustrative

Key takeaways

  • A closed ticket shows that work ended, not that service recovered or why responders acted as they did.
  • Keep two clocks per event: when it happened and when it was written down.
  • Link every action, ruled-out hypothesis and recovery claim to the alert, metric or record behind it.
  • Recovery must be a measured window after the last remediating action; automatic closure never counts.
  • Remove secrets, generalize internal identifiers and describe risky actions by class, and log every redaction.
  • Reject cases that fail a check instead of repairing them into a plausible story.

What should an IT incident training case preserve beyond ticket closure?

It should preserve the timeline: the alerts that fired, what responders believed at each point, the actions they took and who approved them, the runbook version they followed, and the evidence that service actually recovered. A closed ticket records that work ended; the timeline records how the decision was made, which is what an IT incident timeline dataset is for.

The closed ticket is a weak endpoint for another reason. In ServiceNow, for example, a resolved incident can close automatically after a configurable number of days set by a system property, so a Closed state can mean nobody touched the record again rather than that anyone confirmed the fix. Other tools behave similarly. Status fields are workflow bookkeeping, not proof of recovery.

The case structure below rebuilds each incident from linked evidence instead: alert payloads, responder notes, change records and the metric that shows the service came back. It is designed to keep the reasoning while leaving out secrets and operational detail that would be dangerous outside the company.

Why the order of events carries the lesson

Incident decisions only make sense in sequence: a responder who restarted a service at minute 12 knew less than the one who wrote the postmortem a week later. If the dataset flattens the incident into a summary written with hindsight, a model learns to reason from facts nobody had at the time.

Mature incident practices already treat the timeline as the core artifact. PagerDuty's public incident response documentation gives the Scribe role the job of keeping a timeline of key events during an incident, including the result of decisions such as whether to proceed with a rolling restart. Its postmortem template asks for the time the contributing factor began, the time of the page, the time of significant actions and links to the tools or logs that show how each timestamp was established. Google's SRE book example postmortem describes its timeline as the screenplay of the incident, built from the incident management timeline plus other relevant entries.

Those practices give a dataset two things it needs: a record of belief and action as they happened, and a pointer from each timeline line to the evidence behind it. The schema keeps both.

What the linked alert, action and recovery case schema includes

One case is one incident, made of eight linked record types. Every record carries the incident ID, a timestamp in UTC and a source reference, so any line can be traced back to the system that produced it. The table is the asset; field names are suggestions, the joins and checks are not optional.

  • Store two clocks per event, when it happened and when it was written down, because notes typed after the call are common and are not the same evidence.
  • Keep severity at open and at close separately; a downgrade mid-incident is itself a decision.
  • Record the schema version and date (for example 1.0, 2026-10-18) with every export.
What the linked alert, action and recovery case schema includes
RecordRequired fieldsWhat it provesValidation check
Incident headerincident_id, source ticket IDs, severity at open and at close, service, declared_at, resolved_at, closed_at, close method (manual or automatic)Which incident this is and how its status endedTicket IDs resolve in the source system; close method is recorded, not inferred
Alert evidencealert fingerprint, alert name, status (firing or resolved), starts_at, ends_at, labels kept after redaction, link to generating ruleWhat the monitoring system saw and whenEvery alert's start falls before the first responder action that cites it
Timeline eventevent_id, occurred_at, recorded_at, actor role, channel (pager, chat, call, ticket), event text, evidence linkWhat responders knew and said at each momentrecorded_at is not earlier than occurred_at; events are not backfilled without a flag
Hypothesishypothesis_id, stated_at, text, status (open, confirmed, ruled out), evidence for statusWhat responders believed and how that changedA hypothesis marked ruled out cites the event or metric that ruled it out
Action and approvalaction_id, taken_at, actor role, action class (restart, rollback, failover, scale, config change, escalate, communicate), approver role, change record IDWhat was done, under whose authorityActions that need approval under your process carry an approver role and a change record
Runbook referencerunbook ID, version or commit, section followed, deviation noteWhich documented procedure guided the action, and where responders departed from itThe runbook version existed at taken_at; deviations are explicit
Recovery evidencemetric or check name, threshold, observed window start and end, result, linkThat the service recovered, measured rather than assertedRecovery window starts after the last remediating action and lasts the agreed observation period
Redaction logfield path, redaction type (removed, masked, hashed, generalized), reason, reviewer roleWhat was taken out, and that someone checked itNo case ships without a redaction log, even when it says nothing was removed

Where do the alert, action and recovery records come from?

Each record type has a natural home system, and the case is assembled by joining them on the incident ID and on time. Alerting tools export structured payloads, chat and call notes hold the reasoning, change management holds approvals, and monitoring holds the recovery signal.

Prometheus Alertmanager's webhook payload is a good model of what alert evidence should look like. Each alert in it carries a status of firing or resolved, labels, annotations, a start time and end time in RFC 3339 format, a link identifying the entity that caused the alert, and a fingerprint that identifies the alert. That fingerprint is what lets a dataset tell a repeated alert from a new one, which matters for teaching triage under noise.

Expect the reasoning to be the thinnest layer. Chat channels and call notes are where responders say what they suspect and why they chose one action over another, and they are also where secrets, customer names and personal remarks appear. Capture them, but only through the redaction steps described below.

  • Alerting and on-call tools: alert payloads, pages, acknowledgments, escalations.
  • Ticketing or ITSM: incident record, state changes, linked problem and change records.
  • Chat and call notes: hypotheses, decisions, handoffs, Scribe entries.
  • Change management and version control: approvals, deployed versions, rollbacks, runbook versions.
  • Monitoring and status pages: recovery metrics, customer-facing updates.

What evidence explains the IT decision beyond a closed ticket or a final configuration?

The decision is explained by three linked pieces: the hypothesis responders held, the alert or metric that supported it, and the action they chose with the alternatives they rejected. A final configuration file shows the end state but not why the team got there, and the ticket's resolution note is usually written after the fact by one person.

PagerDuty's postmortem process asks that every timeline item point to a metric, graph, search or other page showing where its data came from. Apply the same rule to hypotheses and actions in the dataset: if a responder wrote that disk latency looked normal, the case should link to the panel or query result they were looking at, or mark the claim unsupported.

Ruled-out hypotheses are especially valuable. They record the checks a skilled responder runs before acting, and they are exactly what disappears when an incident is summarized. Keep them with the evidence that ruled them out, and keep postmortem language blame-free, as PagerDuty's process requires, so individual responders are described by role and action rather than judged.

An illustrative checkout outage, recorded well and recorded badly

The case below is illustrative: a fictional software company, an invented checkout service and made-up identifiers and times. None of it comes from a SourceX client. The middle column passes every check; the right column shows a variant that should fail, and why.

An illustrative checkout outage, recorded well and recorded badly
RecordValid case (illustrative)Failing variant (illustrative)Check that catches it
Alert evidenceError-rate alert on checkout, firing at 09:02 UTC, resolved at 09:41Alert start recorded as 09:20, after the restart that cites it at 09:12Alert start must precede the action that cites it
Timeline event09:05 on-call engineer: suspect bad deploy from 08:55; recorded 09:05Same note recorded at 14:00 with no backfill flagBackfilled events must be flagged
HypothesisDatabase saturation ruled out at 09:09, linked to query latency panelDatabase saturation ruled out with no linked evidenceRuled-out hypotheses need evidence
Action and approval09:14 rollback to previous release, approved by incident commander, change record CHG-88Rollback with no approver on a service where your process requires oneApproval required by process must be present
Runbook referenceCheckout rollback runbook, version at commit a1b2c3, step 4 skipped with noteRunbook version committed the following weekRunbook version must exist at action time
Recovery evidenceError rate under the alert threshold for the agreed 30-minute window from 09:20Recovery claimed from the ticket closing automatically three days laterRecovery needs a measured window after the last action
Redaction logTwo database hostnames generalized, one access token removed from a pasted log, reviewer: securityNo redaction log; a session token visible in a chat excerptRedaction log required; secret scanner must pass

Which incidents are worth structuring first?

Start with incidents where all four evidence layers survive: the alert payload, live responder notes, a change or approval record, and a recovery metric for the window afterward. Incidents missing the notes or the recovery signal can still be counted, but they make poor training or evaluation cases because the decision and the outcome are both guesses.

At a glance

Which incidents are worth structuring first

Retention usually decides this before anyone else does. Chat workspaces, alerting tools and monitoring systems keep data for different periods, so an incident from three years ago may have its ticket but not its metrics. Inventory the retention of each source system first, then pick the years where all layers overlap.

Keep some unglamorous cases. Alerts that fired and resolved without action, incidents downgraded after triage and escalations that turned out to be someone else's system teach a model when not to act, which is as useful as a dramatic recovery. Label them so they are not mistaken for incomplete records.

  • Include: incidents with alert, notes, approval and recovery evidence that all resolve.
  • Include, labeled as no action: alerts triaged and closed without remediation, with the triage reasoning recorded.
  • Hold back: incidents where the recovery metric has aged out of retention; mark the outcome unverified.
  • Exclude: security incidents under investigation, incidents tied to legal hold, and incidents dominated by one customer's confidential data.

How to keep secrets and dangerous detail out of the case

Scan and redact before a case leaves the operations team, and treat incident records as sensitive by default. NIST SP 800-61 Revision 3 recommends protecting the confidentiality and integrity of incident response records and restricting access to authorized personnel, because they can describe exploited weaknesses and the actions of individual users.

OWASP's Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than recorded directly, including session identifiers, access tokens, authentication passwords, database connection strings, encryption keys and sensitive personal data. Incident chat is where these leak, because responders paste log lines and command output under time pressure. The same list is a good starting point for a redaction pass over incident cases.

Checklist · 5 items

  1. Remove secrets outright; masking a token still reveals its format and position. Rotate any credential found in an archive, separately from the dataset work.
  2. Generalize internal hostnames, IP ranges and account IDs to stable placeholders, so the timeline still reads coherently.
  3. Replace personal names with roles, and drop personal remarks unrelated to the incident.
  4. Exclude security incidents that are under investigation, involve unpatched weaknesses or carry legal hold; those belong to the security team, not the dataset.
  5. Describe actions by class and outcome rather than exact commands when the commands would work as instructions against your environment.

0 of 5 ticked. Tick each item as you go.

How to validate a case and bound what it may claim

Run the checks in a fixed order and reject a case on the first failure, logging the reason. Repairing a broken case by hand tends to replace the record of what happened with a plausible story, which defeats the purpose.

At a glance

How to validate a case and bound what it may claim

Most checks are joins and time comparisons and run in a short script over exported tables. Publish the validator output alongside the cases so a reviewer can reproduce the pass and failure counts.

Bound every outcome by its evidence. A case can claim that a metric returned below a threshold for a stated window after a rollback; it cannot claim the rollback was the root cause unless the postmortem established that with evidence, and it should not claim the incident never recurred unless the observation period covered it.

This matters when incident cases are used to evaluate AI agents, because an agent that proposes the same rollback should score well only if the case proves the rollback worked. Label every outcome field with its evidence type (measured, stated in postmortem, or unverified) so evaluators can weight it accordingly.

Checklist · 6 items

  1. Identity: incident, alert, event, action and hypothesis IDs are unique, and every record carries the incident ID.
  2. Chronology: alert start before citing action, occurred_at before or equal to recorded_at, and declared_at before resolved_at.
  3. Evidence: every action, every ruled-out hypothesis and every recovery claim links to an evidence record that resolves.
  4. Authority: actions in classes your process requires to be approved carry an approver role and a change record.
  5. Recovery: a measured window follows the last remediating action; automatic closure alone never counts.
  6. Safety: a secret scanner and a personal data scan both pass, and the redaction log is present and reviewed.

0 of 6 ticked. Tick each item as you go.

If you are weighing whether incident history could be licensed

Companies that have run on-call for years often hold thousands of incidents, but only the ones that pass a schema like this are useful outside the company. SourceX can take the incidents that pass through rights review, contracting, data preparation and delivery, so they reach AI developers as a licensed dataset.

Whether a company qualifies is settled first in a brief assessment, without any data leaving its systems. Scope and terms are approved and the agreement signed before any record is delivered, so security incidents, particular services or whole years can be excluded up front.

How often does a published incident report record the decision behind an action?

Rarely, even when the team's own process asks for it. The UK Ministry of Justice Cloud Platform team publishes its incident reports in a public GitHub repository; at commit 8a52e24 (read on 2026-10-27), 39 reports covering incidents from February 2020 to March 2026 hold 341 lines stamped with a clock time. Only 4 of those lines, in 3 reports, use a decision word such as decided, decision or agreed, while 39 carry a remediation verb such as restart, revert, restore or scale. Of the 4 decision lines, 1 says who decided, 1 cites evidence on the same line and none names an option that was turned down.

The missing decisions are not a policy gap. The team's incident process makes the incident lead responsible for making "the final decision whenever we need to choose a course of action" and lists decisions among the items the scribe's log must include, next to important events, discussion topics, actions and results. The log's form is left to the scribe, provided key events are timestamped. Where a choice was logged, then, it sits in the scribe's notes and the incident thread, not in the report a buyer is most likely to be offered.

Some decisions do reach the report, but as untimed prose. The report on the CircleCI security incident of January 2023 explains that a preliminary investigation found service accounts shared between namespaces, then states: "We made the decision that we need to rotate any and all secrets used within the cluster." It gives no time, no decider and no record of what else was weighed, so the case shows the outcome of a judgment without the moment at which it was made.

How do you rebuild a missing decision without writing reasons nobody recorded?

Anchor every field to a line in the record and write not recorded wherever no line exists. The reconstruction is an audit, not a narrative: a reviewer marks each field as recorded, adjacent (supported by a neighboring line) or absent, and an absent field stays empty even when the answer seems obvious in hindsight.

Etsy's debriefing facilitation guide shows what live questioning would have captured. When someone explicitly made a choice, it advises asking about the options they considered and what brought them to that decision, including whether they have ever not made it in similar circumstances. An archive cannot be asked those questions later, so the audit records that they went unanswered instead of guessing. The missing-decision audit checks eight things:

  • Decision moment: the time of the line that commits to a course, or, if none does, the window between the last evidence line and the first action after it, labeled inferred.
  • Trigger: the line that made a choice necessary, cited by its time.
  • Options open: only options the record names; options a reader can think of today are not added.
  • Evidence in view: lines stamped before the decision moment; anything recorded later is tagged hindsight.
  • Decider and authority: the role named on or before the line, and whether that role existed yet when the decision was made.
  • Stated purpose: the purpose clause quoted as written, not rewritten into a cleaner reason.
  • Action and check: the action that followed and the first line that tests whether it worked.
  • Usability: a decision with no recorded options or decider can serve as context for a case, not as its gold label.

What does a reconstructed decision look like on a real incident report?

It looks like a partly filled form, and the empty fields are the finding. The Ministry of Justice report of 11 January 2023, on clusters that could not pull container images after a DockerHub password change, is one of the 3 reports with timed decisions. The reconstruction below uses only the published text; the team that wrote the report has not reviewed it, and OOH in the report means out of hours:

What does a reconstructed decision look like on a real incident report?
Audit fieldDecision at 15:02Decision at 15:36Status
Trigger15:01: the Concourse plan for the password update shows the launch template will change, suggesting node recycle15:28: confirmation that the update will force node recycles across the live and manager clustersRecorded on the line before each decision
Choice as writtenUpdate the password in the live-2 cluster "to determine whether a node recycle will be required"Restore the previous DockerHub password, "to allow the team to manage a clean rotation OOH"Recorded
Options openNot recordedNot recorded; continuing the rotation in working hours is implied by the purpose clause but never statedAbsent
DeciderNot recorded; no incident lead existed until the incident was declared at 15:17Not on the line; the incident lead and scribe roles were taken at 15:17Absent (role adjacent for 15:36)
Action and checkNo line reports the live-2 result; at 15:51 the report notes live-2 was left with an invalid dockerconfig and no impact on users15:40 password changed back; 15:50 image pulling observed working againRecorded

How would a buyer use incident decision context in training or evaluation?

As decision-point tasks frozen at the moment of choice. For the 15:36 decision, a buyer can hand a model every line up to 15:35, ask it to choose between pressing on with the rotation and restoring the old password, and grade the answer against the recorded choice and the 15:50 check. Because options and decider are absent, the case can test whether a model reaches the recorded choice, not whether it weighed the same alternatives or respected the same authority.

Those limits belong in the request. A buyer asking for incident decision context should ask for the scribe's log and the incident thread behind each report, with the eight audit statuses attached per decision and a count of decisions found only in untimed prose, and should treat absent fields as missing data rather than as evidence that nobody deliberated.

Sources for this section

  • The Ministry of Justice Cloud Platform team's public incident archive (39 incident reports at commit 8a52e24, incidents from February 2020 to March 2026) records key events and a context or timeline section per incident; in those reports, 4 of 341 clock-stamped lines use a decision word, under the counting rules stated in the writer notes. Source
  • The team's incident process makes the incident lead responsible for the final decision whenever a course of action must be chosen, lists decisions among the items the scribe's log must include, and leaves the log's form to the scribe provided key events are timestamped. Source
  • The report of 11 January 2023 records decisions at 15:02 and 15:36, the incident declaration and role assignment at 15:17, the password restored at 15:40 and image pulling observed working at 15:50. Source
  • The CircleCI security incident report of January 2023 states the decision to rotate any and all secrets used within the cluster in untimed prose after describing service accounts shared between namespaces. Source
  • Etsy's Debriefing Facilitation Guide advises that when someone explicitly made a choice or decision, the facilitator asks about the options they considered and what brought them to that decision. Source

Frequently asked questions

What is an IT incident timeline dataset?

It is a set of incident cases in which each alert, responder note, hypothesis, action, approval and recovery measurement is a separate timestamped record linked by the incident ID. Unlike a ticket export, it preserves the order in which people learned things and acted, plus the evidence behind each step.

Is a postmortem enough on its own?

A postmortem is a strong source but it is written with hindsight. Use it for confirmed causes and action items, and keep the in-the-moment timeline events, hypotheses and alerts as separate records so the case shows what responders knew at each point.

How do we handle incidents that closed automatically?

Record the close method in the incident header and do not treat automatic closure as proof of recovery. If no measured recovery window exists, label the outcome unverified or leave the case out of any evaluation set.

Can security incidents be included?

Usually only after the security team has closed the investigation, any weakness involved is fixed and counsel has confirmed no legal hold or notification issue applies. Many teams leave security incidents out entirely and use operational incidents instead.

How should responder identities be stored?

Use roles such as on-call engineer, incident commander and Scribe, not names. Keep postmortem language blame-free, remove personal remarks, and check employment policies before any chat content leaves the company.

Sources

  • PagerDuty's incident response documentation says the Scribe maintains a timeline of key events during an incident, including the result of decisions. Source
  • PagerDuty's postmortem template asks the timeline to include when the contributing factor began, the page, significant actions, and links to the tools or logs behind each timestamp. Source
  • PagerDuty's postmortem process asks that each timeline item identify the metric or page its data came from, and describes postmortems as blame-free. Source
  • Google's SRE book example postmortem describes the timeline as the incident's screenplay, starting from the incident management timeline. Source
  • Prometheus Alertmanager's webhook payload gives each alert a firing or resolved status, labels, annotations, RFC 3339 start and end times, a generator URL and a fingerprint. Source
  • NIST SP 800-61 Revision 3 recommends protecting the confidentiality and integrity of incident response records and restricting access to authorized personnel. Source
  • OWASP's Logging Cheat Sheet lists session identifiers, access tokens, passwords, connection strings, encryption keys and sensitive personal data among items to remove, mask, hash or encrypt rather than log. Source
  • ServiceNow documents that resolved incidents close automatically after a number of days set by a system property, which administrators can change. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies