Privacy and preparation
IT runbook exception data: what the engineer did after a step failed
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
IT runbook exception data should record a failed step as its own decision: the failure signal and the state it left, the evidence the engineer checked, one recovery move from a fixed list such as retry, roll back, escalate or hold, who approved it, and an outcome bounded by monitoring. Rebuild the branch from timestamped chat, logs and change records, and replace secrets and live targets with typed placeholders.
License terms
IT runbook exception data
Key takeaways
- The interesting part of an incident is the branch after a step failed, which a closed ticket usually hides.
- Label each exception with one recovery move from a closed list, or unknown when no record shows the branch.
- Rebuild the chronology from timestamped sources before annotating, and describe system conditions rather than blaming people.
- Replace secrets and live targets with consistent typed placeholders, and report any live secret for rotation.
- Hold back records from active security incidents or unpatched vulnerabilities, and label improvised fixes as off-runbook.
How should IT training data capture an engineer's response when a runbook fails?
Capture the failure as its own decision record: the step that failed and the state it left behind, the evidence the engineer checked, the recovery move chosen from a fixed list, who approved it, and an outcome bounded by what monitoring actually showed. Replace secrets, hostnames and other live targets with typed placeholders so the reasoning survives and the operational detail does not.
IT runbook exception data is the record of what happened after a documented procedure stopped working. The runbook itself shows the intended path. The exception record shows the branch: step four returned an error, the engineer read two dashboards and a log, decided not to retry, rolled back step three and asked the platform team for help. That branch is where operational judgment lives, and it is the part most ticket histories compress into a single line saying resolved.
This guide gives a failure recovery and evidence annotation scheme you can apply to incident records, a worked example, the exception where leaving the runbook was right, and the records that should not be used at all.
What evidence explains the decision beyond a closed ticket?
A closed ticket and a final configuration show where the system ended up, not which branch the engineer took or why. The explanation lives in the chronology between the alert and the close: chat messages, command or automation logs, change records and the postmortem timeline.
Automation tools make the branch explicit, which is a useful model for annotating human work. In AWS Systems Manager Automation, a step's onFailure setting defaults to Abort, can be set to Continue to move on to the next step, or can name a specific step to jump to. In Ansible, a block can carry a rescue section that runs only after a task returns a failed state and an always section that runs regardless; ignore_errors lets a play continue after a failed task but does not cover connection failures or syntax errors, and failed_when lets an author define what counts as failure.
A human runbook rarely declares any of this. The engineer decides on the spot whether to abort, continue, retry or jump elsewhere, so the annotation has to recover that choice from the evidence. If no record shows which branch was taken, label the move unknown rather than guessing from the end state.
Which recovery move did the engineer make, and how is each one mislabeled?
Label every exception with one primary recovery move from a closed list. A fixed list keeps annotators consistent and lets a model learn the decision rather than the prose style of whoever wrote the ticket.
| Recovery move | What it means | Evidence that shows it | Common mislabel |
|---|---|---|---|
| Retry unchanged | Ran the same step again with the same inputs | Repeated command or automation run with identical parameters | Recorded as fixed when the second try simply passed |
| Retry with changed input | Ran the step again after changing a parameter, target or timing | Diff between the two runs | Recorded as a retry, hiding the change that mattered |
| Skip and continue | Accepted the failure and moved to the next step | Later steps run with the failed step left unresolved | Not recorded at all |
| Roll back | Reversed earlier steps to the last known good state | Revert command, change record or deployment history | Recorded as resolved |
| Switch procedure | Left this runbook for another documented one | Link to the second runbook in chat or the ticket | Recorded as the second runbook's success |
| Escalate or hand off | Passed the decision to another team or role | Page, chat mention or reassignment with the context sent | Recorded only as a reassignment |
| Stop and hold | Paused changes and declared or raised an incident | Incident declaration or change freeze message | Recorded as abandoned |
| Improvise off-runbook | Took an undocumented action | Commands with no matching runbook step | Folded into the runbook steps as if documented |
Which fields should each exception record carry?
Annotate each exception with the fields below. One record covers one failed step and the response to it; an incident with three failed steps produces three linked records sharing an incident id.
At a glance
Which fields should each exception record carry
Write the rationale field as a short paraphrase of what the evidence shows the engineer weighed, never as an invented quote, and keep every value traceable to a timestamped source record.
| Field | Allowed values or format | Source of truth | Annotator rule |
|---|---|---|---|
| incident_ref and runbook_ref | Stable ids plus runbook version or commit | Incident tool and runbook repository | Use the version in force at the time, not today's |
| failed_step | Step number and its documented intent | Runbook text | Quote the step intent, not its commands |
| failure_signal | Error class, exit status or symptom | Command output, automation log or alert | Generalize messages that reveal internal names |
| failure_class | Precondition unmet, permission, dependency down, timeout, unexpected state, tooling defect | Annotator judgment from evidence | Pick one; record a second only with evidence |
| evidence_consulted | List of evidence types checked, with timestamps | Chat, dashboards, logs | Only what the record shows was opened or cited |
| recovery_move | One of the eight moves | Chronology | Unknown if no record shows the branch |
| approval | Role that approved and when, or none required | Chat, change record, incident roles | Role labels, not names |
| actions_taken | Ordered actions with typed placeholders | Command and automation logs | No secrets, no live hosts, no exploit detail |
| bounded_outcome | What monitoring showed and over what window | Metrics, alerts, follow-up tickets | Never infer success from a closed status |
| runbook_change and sensitivity | Edit made or requested; sensitivity flags raised | Runbook history, review notes | Link the change; route flagged records to review |
How do you rebuild the chronology before annotating?
Rebuild the sequence from timestamped sources first, then annotate. Order the alert, the chat, the command or automation logs and any change records on one timeline, and only then decide what the engineer saw at each point.
Incident practice already produces most of this if someone kept the timeline. PagerDuty's public incident response documentation describes a scribe role whose purpose is to maintain a timeline of key events, documenting actions and follow-up items, and it asks scribes to note actions the incident commander assigned and the result of decisions polled on the call. Its postmortem guidance asks for an accurate timeline and warns against alternate-reality statements that mix what happened with what might have prevented it.
Annotate blamelessly. Google's SRE book describes a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes and the follow-up actions, and says it should focus on contributing causes without indicting any individual or team. PagerDuty's guidance similarly cautions against the label human error. In the scheme, that means failure_class describes the system condition, such as a permission that changed, never the person who hit it.
What should be stripped, and what kept as a typed placeholder?
Remove anything that grants access or would help someone attack the environment, and replace it with a placeholder that keeps its type and role in the sequence. A model learns as much from ROLE_DEPLOY lacked permission on LB_LISTENER_1 as from the real names, and the record stops being a map of your network.
The OWASP Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than recorded directly, including session identifiers, access tokens, authentication passwords, database connection strings, encryption keys and commercially sensitive information. The OWASP Secrets Management Cheat Sheet adds the reason redaction is not enough on its own: stolen credentials remain usable until they expire or are revoked. If a live secret turns up in a source record, report it for rotation as well as removing it.
| Found in the raw record | Treatment | Placeholder example |
|---|---|---|
| Passwords, tokens, keys, connection strings | Remove and report for rotation | SECRET_REDACTED with its type |
| Hostnames, IP addresses, account and resource ids | Replace consistently within the incident | HOST_A, ACCOUNT_1, LB_LISTENER_1 |
| Customer names and identifiers | Replace or drop | CUSTOMER_1 |
| Engineer names and handles | Replace with role | ONCALL_PRIMARY, INCIDENT_COMMANDER |
| Exploit steps or detailed security bypasses | Generalize to the class of action | Applied vendor workaround for a known vulnerability |
| Internal topology and firewall rules | Summarize at the level the decision needs | Dependency between service tier and data tier |
How would an illustrative failed certificate rotation be annotated?
This example is invented to show the scheme; no company, system or person is real, and every count and duration is illustrative. A runbook for rotating a public TLS certificate has six steps. Step three uploads the new certificate; step four attaches it to the load balancer listener.
At a glance
How would an illustrative failed certificate rotation be annotated
Step four fails with an access denied error. The on-call engineer checks the deployment role's recent changes and sees a permissions update merged two days earlier. The old certificate is still valid for 9 more days. Instead of retrying, the engineer pages the platform team, and the incident commander approves holding the rotation and leaving the old certificate attached. Monitoring shows no change in error rates over the following 30 minutes. The platform team restores the permission the next morning and the rotation completes on the first attempt; the runbook gains a precondition check at step one.
- failed_step: 4, attach new certificate to listener
- failure_signal: access denied on LB_LISTENER_1
- failure_class: permission
- evidence_consulted: role change history; certificate expiry date; error-rate dashboard
- recovery_move: stop and hold, with escalate or hand off as the linked second move
- approval: INCIDENT_COMMANDER, before the hold
- bounded_outcome: no error-rate change in a 30-minute window; rotation completed in a later change
- runbook_change: precondition check for listener permissions added at step 1
When is leaving the runbook the right call?
Sometimes the correct response is an undocumented action, and the record should show it as such rather than forcing it into a documented step. Label it improvise off-runbook, record the evidence that justified it and whether anyone approved it, and link the runbook edit that followed if there was one.
Treat these records carefully in a dataset. An improvised fix that worked once is not a recommendation, and a model trained on many of them without the surrounding evidence learns that skipping procedure is normal. Keep the share of improvised records visible in the dataset summary, and include the cases where improvisation made things worse as well as better.
The improvement loop is part of the record too. NIST SP 800-61 Revision 3, finalized in April 2025, organizes incident response around the CSF 2.0 Functions and places lessons learned under the Improvement category of the Identify Function. The runbook_change field is where an exception record connects to that loop.
Which records should be held back or sent to a security reviewer first?
Some exception records should not be used even after redaction. Hold back any record from an incident with an active or suspected attacker, any record whose value depends on a vulnerability that may still be unpatched, and any record where you cannot tell whether a secret has been rotated.
Route others to a security reviewer before inclusion: records mentioning customer environments you manage on someone else's behalf, records that quote vendor support material under confidentiality terms, and records where placeholders would still let a reader infer your architecture. A reviewer can narrow the record to the decision fields and drop the action detail.
When the volume of usable records falls below what a useful set needs after these cuts, say so in the dataset description rather than padding it with synthetic exceptions presented as real.
Who decides whether runbook exception records can be licensed?
Annotated exception records are a strong candidate for training and evaluating operations agents, but annotation does not settle who may license them. Access to the incident tool, the privacy position of names and customer data, ownership of the records and the contractual right to license them are separate questions, and managed service providers in particular should check what their client agreements say about operational records.
An operations team that decides to license annotated incident history can rely on SourceX for the rights review, the contract, preparation and delivery. As a general guide, buyers want two or more years of consistent history and tens of thousands of records in one system, though rare work can qualify with less, deal by deal. The first step is a brief company assessment, and no incident records are uploaded for it.
Frequently asked questions
What is IT runbook exception data?
It is the record of what an engineer did after a documented runbook step failed: the failure, the evidence checked, the recovery move, any approval and the outcome. It differs from runbook text, which shows only the intended steps.
What should the failure recovery annotation scheme include?
At minimum the incident and runbook references with versions, the failed step, the failure signal and class, the evidence consulted, one labeled recovery move, approval, redacted actions, a bounded outcome and any runbook change. Each value should trace to a timestamped source record.
Why not use the closed ticket's resolution notes?
Resolution notes describe the end state and often compress several branches into one line. The recovery decision is visible only in the chronology of chat, logs and change records between the alert and the close.
How do you remove secrets without losing the reasoning?
Replace each secret, host or identifier with a typed placeholder that stays consistent within the incident, so the sequence still reads correctly. Report any live secret you find for rotation, because removing it from the record does not revoke it.
Should improvised fixes be included in training data?
Yes, if they are labeled as off-runbook, carry the evidence and approval behind them, and appear alongside cases where improvisation failed. Without that context a model can learn that skipping procedure is normal.
Sources
- AWS documents that an Automation step's onFailure setting defaults to Abort, can be Continue, or can name a step to go to. Source
- Ansible's blocks guide says rescue tasks run only after a task returns a failed state and always tasks run regardless of the block's result. Source
- Ansible's error handling guide says ignore_errors does not cover connection failures, execution issues or syntax errors, and failed_when defines what counts as failure. Source
- PagerDuty's incident response documentation describes the scribe's purpose as maintaining a timeline of key events, actions and follow-up items. Source
- PagerDuty's postmortem guidance asks for an accurate timeline, blameless language, no human error label and no alternate-reality statements. Source
- Google's SRE book defines a postmortem as a written record of an incident, its impact, actions taken, root causes and follow-up actions, written blamelessly. Source
- The OWASP Logging Cheat Sheet lists data that should usually be removed, masked, sanitized, hashed or encrypted rather than logged. Source
- The OWASP Secrets Management Cheat Sheet notes that stolen credentials remain usable until the backing service expires or revokes them. Source
- NIST SP 800-61 Revision 3 (April 2025) frames incident response as a CSF 2.0 Community Profile and places lessons learned under Identify's Improvement category. Source
See if your company qualifies
A short company assessment. No data uploads are needed.