Skip to content

Privacy and preparation

IT runbook exception data: what the engineer did after a step failed

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

IT runbook exception data should record a failed step as its own decision: the failure signal and the state it left, the evidence the engineer checked, one recovery move from a fixed list such as retry, roll back, escalate or hold, who approved it, and an outcome bounded by monitoring. Rebuild the branch from timestamped chat, logs and change records, and replace secrets and live targets with typed placeholders.

License terms

IT runbook exception data

This page covers IT runbook exception data: what the engineer did after a step failed.Illustrative

Key takeaways

  • The interesting part of an incident is the branch after a step failed, which a closed ticket usually hides.
  • Label each exception with one recovery move from a closed list, or unknown when no record shows the branch.
  • Rebuild the chronology from timestamped sources before annotating, and describe system conditions rather than blaming people.
  • Replace secrets and live targets with consistent typed placeholders, and report any live secret for rotation.
  • Hold back records from active security incidents or unpatched vulnerabilities, and label improvised fixes as off-runbook.

How should IT training data capture an engineer's response when a runbook fails?

Capture the failure as its own decision record: the step that failed and the state it left behind, the evidence the engineer checked, the recovery move chosen from a fixed list, who approved it, and an outcome bounded by what monitoring actually showed. Replace secrets, hostnames and other live targets with typed placeholders so the reasoning survives and the operational detail does not.

IT runbook exception data is the record of what happened after a documented procedure stopped working. The runbook itself shows the intended path. The exception record shows the branch: step four returned an error, the engineer read two dashboards and a log, decided not to retry, rolled back step three and asked the platform team for help. That branch is where operational judgment lives, and it is the part most ticket histories compress into a single line saying resolved.

This guide gives a failure recovery and evidence annotation scheme you can apply to incident records, a worked example, the exception where leaving the runbook was right, and the records that should not be used at all.

What evidence explains the decision beyond a closed ticket?

A closed ticket and a final configuration show where the system ended up, not which branch the engineer took or why. The explanation lives in the chronology between the alert and the close: chat messages, command or automation logs, change records and the postmortem timeline.

Automation tools make the branch explicit, which is a useful model for annotating human work. In AWS Systems Manager Automation, a step's onFailure setting defaults to Abort, can be set to Continue to move on to the next step, or can name a specific step to jump to. In Ansible, a block can carry a rescue section that runs only after a task returns a failed state and an always section that runs regardless; ignore_errors lets a play continue after a failed task but does not cover connection failures or syntax errors, and failed_when lets an author define what counts as failure.

A human runbook rarely declares any of this. The engineer decides on the spot whether to abort, continue, retry or jump elsewhere, so the annotation has to recover that choice from the evidence. If no record shows which branch was taken, label the move unknown rather than guessing from the end state.

Which recovery move did the engineer make, and how is each one mislabeled?

Label every exception with one primary recovery move from a closed list. A fixed list keeps annotators consistent and lets a model learn the decision rather than the prose style of whoever wrote the ticket.

Which recovery move did the engineer make, and how is each one mislabeled?
Recovery moveWhat it meansEvidence that shows itCommon mislabel
Retry unchangedRan the same step again with the same inputsRepeated command or automation run with identical parametersRecorded as fixed when the second try simply passed
Retry with changed inputRan the step again after changing a parameter, target or timingDiff between the two runsRecorded as a retry, hiding the change that mattered
Skip and continueAccepted the failure and moved to the next stepLater steps run with the failed step left unresolvedNot recorded at all
Roll backReversed earlier steps to the last known good stateRevert command, change record or deployment historyRecorded as resolved
Switch procedureLeft this runbook for another documented oneLink to the second runbook in chat or the ticketRecorded as the second runbook's success
Escalate or hand offPassed the decision to another team or rolePage, chat mention or reassignment with the context sentRecorded only as a reassignment
Stop and holdPaused changes and declared or raised an incidentIncident declaration or change freeze messageRecorded as abandoned
Improvise off-runbookTook an undocumented actionCommands with no matching runbook stepFolded into the runbook steps as if documented

Which fields should each exception record carry?

Annotate each exception with the fields below. One record covers one failed step and the response to it; an incident with three failed steps produces three linked records sharing an incident id.

At a glance

Which fields should each exception record carry

Write the rationale field as a short paraphrase of what the evidence shows the engineer weighed, never as an invented quote, and keep every value traceable to a timestamped source record.

Which fields should each exception record carry?
FieldAllowed values or formatSource of truthAnnotator rule
incident_ref and runbook_refStable ids plus runbook version or commitIncident tool and runbook repositoryUse the version in force at the time, not today's
failed_stepStep number and its documented intentRunbook textQuote the step intent, not its commands
failure_signalError class, exit status or symptomCommand output, automation log or alertGeneralize messages that reveal internal names
failure_classPrecondition unmet, permission, dependency down, timeout, unexpected state, tooling defectAnnotator judgment from evidencePick one; record a second only with evidence
evidence_consultedList of evidence types checked, with timestampsChat, dashboards, logsOnly what the record shows was opened or cited
recovery_moveOne of the eight movesChronologyUnknown if no record shows the branch
approvalRole that approved and when, or none requiredChat, change record, incident rolesRole labels, not names
actions_takenOrdered actions with typed placeholdersCommand and automation logsNo secrets, no live hosts, no exploit detail
bounded_outcomeWhat monitoring showed and over what windowMetrics, alerts, follow-up ticketsNever infer success from a closed status
runbook_change and sensitivityEdit made or requested; sensitivity flags raisedRunbook history, review notesLink the change; route flagged records to review

How do you rebuild the chronology before annotating?

Rebuild the sequence from timestamped sources first, then annotate. Order the alert, the chat, the command or automation logs and any change records on one timeline, and only then decide what the engineer saw at each point.

Incident practice already produces most of this if someone kept the timeline. PagerDuty's public incident response documentation describes a scribe role whose purpose is to maintain a timeline of key events, documenting actions and follow-up items, and it asks scribes to note actions the incident commander assigned and the result of decisions polled on the call. Its postmortem guidance asks for an accurate timeline and warns against alternate-reality statements that mix what happened with what might have prevented it.

Annotate blamelessly. Google's SRE book describes a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes and the follow-up actions, and says it should focus on contributing causes without indicting any individual or team. PagerDuty's guidance similarly cautions against the label human error. In the scheme, that means failure_class describes the system condition, such as a permission that changed, never the person who hit it.

What should be stripped, and what kept as a typed placeholder?

Remove anything that grants access or would help someone attack the environment, and replace it with a placeholder that keeps its type and role in the sequence. A model learns as much from ROLE_DEPLOY lacked permission on LB_LISTENER_1 as from the real names, and the record stops being a map of your network.

The OWASP Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than recorded directly, including session identifiers, access tokens, authentication passwords, database connection strings, encryption keys and commercially sensitive information. The OWASP Secrets Management Cheat Sheet adds the reason redaction is not enough on its own: stolen credentials remain usable until they expire or are revoked. If a live secret turns up in a source record, report it for rotation as well as removing it.

What should be stripped, and what kept as a typed placeholder?
Found in the raw recordTreatmentPlaceholder example
Passwords, tokens, keys, connection stringsRemove and report for rotationSECRET_REDACTED with its type
Hostnames, IP addresses, account and resource idsReplace consistently within the incidentHOST_A, ACCOUNT_1, LB_LISTENER_1
Customer names and identifiersReplace or dropCUSTOMER_1
Engineer names and handlesReplace with roleONCALL_PRIMARY, INCIDENT_COMMANDER
Exploit steps or detailed security bypassesGeneralize to the class of actionApplied vendor workaround for a known vulnerability
Internal topology and firewall rulesSummarize at the level the decision needsDependency between service tier and data tier

How would an illustrative failed certificate rotation be annotated?

This example is invented to show the scheme; no company, system or person is real, and every count and duration is illustrative. A runbook for rotating a public TLS certificate has six steps. Step three uploads the new certificate; step four attaches it to the load balancer listener.

At a glance

How would an illustrative failed certificate rotation be annotated

Step four fails with an access denied error. The on-call engineer checks the deployment role's recent changes and sees a permissions update merged two days earlier. The old certificate is still valid for 9 more days. Instead of retrying, the engineer pages the platform team, and the incident commander approves holding the rotation and leaving the old certificate attached. Monitoring shows no change in error rates over the following 30 minutes. The platform team restores the permission the next morning and the rotation completes on the first attempt; the runbook gains a precondition check at step one.

  • failed_step: 4, attach new certificate to listener
  • failure_signal: access denied on LB_LISTENER_1
  • failure_class: permission
  • evidence_consulted: role change history; certificate expiry date; error-rate dashboard
  • recovery_move: stop and hold, with escalate or hand off as the linked second move
  • approval: INCIDENT_COMMANDER, before the hold
  • bounded_outcome: no error-rate change in a 30-minute window; rotation completed in a later change
  • runbook_change: precondition check for listener permissions added at step 1

When is leaving the runbook the right call?

Sometimes the correct response is an undocumented action, and the record should show it as such rather than forcing it into a documented step. Label it improvise off-runbook, record the evidence that justified it and whether anyone approved it, and link the runbook edit that followed if there was one.

Treat these records carefully in a dataset. An improvised fix that worked once is not a recommendation, and a model trained on many of them without the surrounding evidence learns that skipping procedure is normal. Keep the share of improvised records visible in the dataset summary, and include the cases where improvisation made things worse as well as better.

The improvement loop is part of the record too. NIST SP 800-61 Revision 3, finalized in April 2025, organizes incident response around the CSF 2.0 Functions and places lessons learned under the Improvement category of the Identify Function. The runbook_change field is where an exception record connects to that loop.

Which records should be held back or sent to a security reviewer first?

Some exception records should not be used even after redaction. Hold back any record from an incident with an active or suspected attacker, any record whose value depends on a vulnerability that may still be unpatched, and any record where you cannot tell whether a secret has been rotated.

Route others to a security reviewer before inclusion: records mentioning customer environments you manage on someone else's behalf, records that quote vendor support material under confidentiality terms, and records where placeholders would still let a reader infer your architecture. A reviewer can narrow the record to the decision fields and drop the action detail.

When the volume of usable records falls below what a useful set needs after these cuts, say so in the dataset description rather than padding it with synthetic exceptions presented as real.

Who decides whether runbook exception records can be licensed?

Annotated exception records are a strong candidate for training and evaluating operations agents, but annotation does not settle who may license them. Access to the incident tool, the privacy position of names and customer data, ownership of the records and the contractual right to license them are separate questions, and managed service providers in particular should check what their client agreements say about operational records.

An operations team that decides to license annotated incident history can rely on SourceX for the rights review, the contract, preparation and delivery. As a general guide, buyers want two or more years of consistent history and tens of thousands of records in one system, though rare work can qualify with less, deal by deal. The first step is a brief company assessment, and no incident records are uploaded for it.

Frequently asked questions

What is IT runbook exception data?

It is the record of what an engineer did after a documented runbook step failed: the failure, the evidence checked, the recovery move, any approval and the outcome. It differs from runbook text, which shows only the intended steps.

What should the failure recovery annotation scheme include?

At minimum the incident and runbook references with versions, the failed step, the failure signal and class, the evidence consulted, one labeled recovery move, approval, redacted actions, a bounded outcome and any runbook change. Each value should trace to a timestamped source record.

Why not use the closed ticket's resolution notes?

Resolution notes describe the end state and often compress several branches into one line. The recovery decision is visible only in the chronology of chat, logs and change records between the alert and the close.

How do you remove secrets without losing the reasoning?

Replace each secret, host or identifier with a typed placeholder that stays consistent within the incident, so the sequence still reads correctly. Report any live secret you find for rotation, because removing it from the record does not revoke it.

Should improvised fixes be included in training data?

Yes, if they are labeled as off-runbook, carry the evidence and approval behind them, and appear alongside cases where improvisation failed. Without that context a model can learn that skipping procedure is normal.

Sources

  • AWS documents that an Automation step's onFailure setting defaults to Abort, can be Continue, or can name a step to go to. Source
  • Ansible's blocks guide says rescue tasks run only after a task returns a failed state and always tasks run regardless of the block's result. Source
  • Ansible's error handling guide says ignore_errors does not cover connection failures, execution issues or syntax errors, and failed_when defines what counts as failure. Source
  • PagerDuty's incident response documentation describes the scribe's purpose as maintaining a timeline of key events, actions and follow-up items. Source
  • PagerDuty's postmortem guidance asks for an accurate timeline, blameless language, no human error label and no alternate-reality statements. Source
  • Google's SRE book defines a postmortem as a written record of an incident, its impact, actions taken, root causes and follow-up actions, written blamelessly. Source
  • The OWASP Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than logged. Source
  • The OWASP Secrets Management Cheat Sheet notes that stolen credentials remain usable until the backing service expires or revokes them. Source
  • NIST SP 800-61 Revision 3 (April 2025) frames incident response as a CSF 2.0 Community Profile and places lessons learned under Identify's Improvement category. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies