AI uses for records
Outdated runbook incident data: the step succeeded, the outage grew
By SourceX Editorial · Updated
10 sections
Short answer
Outdated runbook incident data captures incidents that grew while responders followed the procedure as written: the runbook version in force, each step's time beside the impact metric, any deviation and the revision that followed. Public records show the pattern and the gap. Of 356 revisions to Grafana Mimir's runbook, 9 warn that a step can cause harm, and none of their commit messages mentions an incident.
Record timeline
Outdated runbook incident
Key takeaways
- A step that runs without error can still deepen an incident when the system no longer matches an assumption in the procedure.
- Only step timestamps aligned with a finely sampled impact metric can separate a harmful step from an incident that was worsening anyway.
- Amazon's 2017 S3 outage and Knight Capital's 2012 trading incident both turned on a documented or standard action whose precondition nobody checked.
- In 356 revisions of Grafana Mimir's runbook, 216 rewrite existing instructions, yet only 2 commit messages tie a change to a specific incident.
- A useful dataset labels coincident worsening and deviations too, so a model does not learn to abandon procedures whenever a metric rises.
What does it mean when following the runbook made the incident worse?
It means a responder carried out a documented step as written, the step ran without error, and the impact grew because the system no longer matched something the step assumed. Nothing failed that a responder could react to, so these cases are easy to file as bad luck and worth recording with care.
The assumption can be about capacity, replication, topology, software versions or code that was supposed to be dormant. It can also be a parameter the step accepts without checking that it stays inside a safe range. In each case the procedure is the cause, not the person, so the fitting remedy is a change to the runbook rather than a note about the responder.
This is a different record from three neighbors on this site. The runbook exception page covers a step that failed and the branch the engineer took next. The incident timeline page covers keeping alerts, beliefs and actions in order. The root cause uncertainty page covers grading competing explanations. Here the question is narrower: which records let a buyer label a case as the procedure itself made it worse, and tell it apart from an incident that would have grown anyway.
Two public incidents where the documented response deepened the outage
Two well-documented public incidents show the pattern: a step taken from an established procedure, executed by authorized staff, that turned a contained problem into a large one, followed by a change to the tool or procedure. Both are years old, and both are useful because a detailed public account exists: Amazon's own summary in one case and the SEC's findings in the other.
On February 28, 2017, according to Amazon's post-event summary, an authorized S3 team member using an established playbook ran a command meant to remove a small number of servers from a subsystem used by the S3 billing process. One input was entered incorrectly and a larger set of servers was removed, taking capacity from the index and placement subsystems, which then needed a full restart. The index subsystem served most requests again by 12:26 PM PST and fully recovered by 1:18 PM; placement finished at 1:54 PM. Amazon said it changed the tool to remove capacity more slowly, added safeguards that block removals taking a subsystem below its minimum required capacity, and audited other operational tools for similar checks.
The second case is a response rather than a planned task. In its 2013 order against Knight Capital, the SEC describes how on August 1, 2012, after the problem was noticed, Knight removed newly deployed code from the seven servers where it had been installed correctly, and that action worsened the problem because incoming orders then triggered old Power Peg code on those servers. The SEC's press release says the router sent more than 4 million orders in the first 45 minutes while trying to fill 212 customer orders. A rollback is a standard move; this one rested on the stale belief that the old code was harmless.
Read as data, the two cases share a structure: a documented action, a precondition nobody checked at the moment of execution, a measurable jump in impact after the action, and a later change that encodes the missing check. Those four parts are what a buyer should ask a supplier's records to show.
Six labels for why impact grew after a step
Label each case by what the records show about why impact grew after the step, and include the two labels that clear the runbook. Without them, every incident that worsened during a procedure gets blamed on the procedure, and a model learns to abandon runbooks whenever a metric moves.
The rubric below gives each label, the evidence that supports it, a public example where one exists and how the case should be used. The first four blame the procedure in different ways; the last two do not:
| Label | What the records show | Evidence required | Public example | Use in training or evaluation |
|---|---|---|---|---|
| Stale assumption | The step was correct for the system the runbook was written for; the system changed | Runbook version and date beside the change record that invalidated it | Mimir runbook, 2026: the no-data-loss rule for a corrupted block stopped holding under block-builder | Positive examples of checking a precondition before acting |
| Unsafe input or scope | The step accepts a value that can exceed a safe bound | Command log with the input, and the bound the revision later enforced | Amazon S3, 2017: one mistyped input removed more servers than intended | Items that test whether a model checks magnitude before executing |
| Standard response met hidden state | A rollback, restart or failover interacted with state the runbook never mentions | Timestamps of the response and the state it exposed | Knight Capital, 2012: removing new code exposed dormant old code | Cases that teach why a rollback needs its own preconditions |
| Wrong order or missing stop | A step is safe only after a check listed later, or never | Revision diff that moves the step or adds the check | Mimir runbook, 2026: the manual forget step moved last, with a data loss caution | Contrast pairs of the old and revised order |
| Coincident worsening | Impact was already rising at the same rate before the step | Impact series with its trend before and after the step | None in the public cases above | Negative examples: do not blame the procedure |
| Deviation, not the runbook | The responder left the procedure and the departure caused the harm | Deviation record with time, reason and approver | See the runbook exception page | Route to exception data, not to runbook revision labels |
Which records separate a harmful step from bad timing?
Step-level timestamps lined up against a finely sampled impact metric separate a harmful step from bad timing, because the test is whether the trend changed when the step ran. A closed ticket, a final configuration or a postmortem paragraph cannot do it; they say the incident got worse, not when relative to each action.
Three time series do the work. The first is the execution record for each step: when it started and ended, from command history, automation run logs or chat. The second is the impact metric the incident was judged by, such as error rate, queue depth or orders sent, at a resolution finer than the gap between steps. The third is the change log for the system the runbook targets, which shows when an assumption stopped being true, for example a deploy that changed replication or a capacity reduction.
Two records turn the series into a label. One is the deviation log: whether responders stopped following the runbook, at what time, on whose call and why. The other is the escalation signal: the first moment a reading contradicted the runbook's assumption and could have been seen, which lets a grader ask whether stopping earlier was possible. The figure shows the core comparison with illustrative values; the same step looks harmful in one series and innocent in the other.
What 356 revisions of a public runbook show about where lessons land
When the lesson from a worsened incident concerns the procedure, the fix belongs in a runbook revision, and public history shows how rarely that revision says what prompted it. Grafana Mimir's alert runbook, maintained since April 2020 and renamed four times, has 356 revisions through October 27, 2026; most change existing instructions rather than add new ones, and the few that add a warning about a step's side effects never mention an incident.
Three revisions show the categories the rubric needs. In August 2020, a recovery command for uploading blocks gained a dry-run flag and a warning against the -d option, labeled dangerous, "because it will delete from the bucket any block which is not in the local filesystem." The commit message reads Improved doc based on feedback received. In May 2026, a pull request corrected guidance that had treated a corrupted level-1 block as safe because the data is replicated; its description says the old text ignored that block-builder deployments do not replicate those blocks, so such corruption must be seen as potential data loss, a stale assumption in plain words. In July 2026, a revision moved the manual forget step for unexpected ring members to the end, added a check for instances left after scale-down, and warned that forgetting an instance that still runs or owns data can lead to data loss.
Read as labels, the diffs are excellent and the reasons are thin. Six of 356 commit messages contain an incident word, and two tie a change to a specific incident the author describes. For buyers, the revision history supplies what changed; the link to the incident that taught it has to come from the incident tracker. The counts and rules:
| Measure | Count | Rule applied |
|---|---|---|
| Revisions of the runbook file | 356 | Non-merge commits touching any of its five paths, 2020-04-09 to 2026-10-27 |
| Rewrite or delete existing text | 216 (60.7%) | At least one removed line not re-added in the same commit |
| Only add text | 134 | No net removed lines; 6 more change no text once blank lines, trailing spaces and moved lines are ignored (2 renames) |
| Flagged by the caution keyword filter | 48 | An added line contains a word such as warning, careful, dangerous, never or data loss |
| Warn that a step can cause harm | 9 | One reader's label on the 48; 8 of the 9 edit existing text |
| Message ties the change to a specific incident | 2 | One reader's label on the 6 messages with an incident word; none of the 9 warnings |
The step-impact trace: fields a worsened-incident record needs
A worsened-incident record is usable when a reader can line up the procedure, its execution and the impact on one clock, and then see the revision that followed. The fields below extend an ordinary incident timeline with the parts specific to this question; every timestamp should be in UTC and every value traceable to a source record:
Steps
The step-impact trace: fields a worsened-incident record needs
Checklist · 8 items
- Runbook identity: the document and version in force when the step ran, with its last revision date.
- Step execution: step number, start and end time, the exact inputs, and who or what ran it, with people replaced by role tokens.
- Impact series: the metric the incident was judged by, sampled at least as often as steps were executed, from 30 minutes before the first step to recovery.
- Assumption check: the precondition the step relied on, stated in a sentence, and the change record showing when it stopped holding, if it did.
- Escalation signal: the first reading that contradicted the precondition and the time a responder could have seen it.
- Deviation record: whether and when responders stopped following the runbook, who decided and the stated reason.
- Revision link: the later runbook change, its diff and the incident identifier it cites, or a coded value saying no link exists.
- Label: one value from the six-label rubric, the evidence pointer behind it and the labeler's role.
0 of 8 ticked. Tick each item as you go.
How buyers would use worsened-incident records
Buyers use these records to teach and test one judgment: whether to execute the next documented step, hold, or escalate when the evidence suggests the procedure's assumption is wrong. Operations agents that read runbooks and run commands need that judgment more than they need more examples of runbooks followed successfully.
For evaluation, give a model the runbook version in force, the telemetry up to the moment before a harmful step and the change log, and ask what it would do next. Grade it against the later revision and the step-aligned impact: a model that runs the step anyway reproduces the incident; one that names the violated precondition and holds or escalates matches what the revision encoded. Include coincident-worsening cases and ordinary successful runs in the same set, so a model cannot score well by stopping every time a metric rises.
For training, the strongest material is a contrast pair: the step as written before the incident, the step as revised after it, and the evidence that connects them. The Mimir examples show what such pairs look like in public; company archives can supply the incident record that public runbooks leave out.
The strongest objection: the incident would have worsened anyway
Sometimes it would have, and the records have to be able to say so. An incident that was already growing at a steady rate before the step, and kept that rate afterward, is coincident worsening, and labeling it as a harmful step teaches a model to distrust procedures that were working.
Two checks keep the label honest. Compare the impact trend before and after the step against its own variability, and look for similar incidents where the same step was run or skipped. Hindsight is the other trap: the precondition that looks obvious in the postmortem may not have been visible at the time, which is why the escalation signal field records when it could first have been seen, not when it was understood.
A third limit is selection. Public cases like Amazon's and Knight's are published because they were severe, and runbook revisions are written when someone took the time. A dataset built only from famous outages and edited runbooks will overstate how often procedures cause harm; the denominator should be every execution of the same runbook in the window, harmful or not.
What company archives add, and how to scope a request
Company archives add the parts public sources almost never keep together: command and automation logs with timestamps, the incident channel, the metrics export for the incident window, and runbook pull requests that cite incident identifiers. Joined, they produce the step-aligned record that neither a postmortem nor a revision history provides alone.
They also carry risk. Command logs hold hostnames, account identifiers and sometimes secrets; incident channels name employees and customers; some runbooks describe security controls in detail. Secrets come out and get rotated, infrastructure names become consistent placeholders, people become role tokens, and incidents involving an attacker or an unpatched weakness stay out until a security reviewer clears them.
SourceX does not keep incident or runbook records in stock; it sources each dataset from businesses to match a buyer's request, which sets out scope, volume, language, format, timing and licensing needs. A request for this kind of data should also name the rubric labels needed, the minimum trace fields, the impact metric resolution and an acceptance test, for example the share of records with a revision link or a coded no-link value. Whether a company's history fits is decided case by case.
Methods note: sources, counting rules and limits
The runbook counts come from a script that runs git log and git diff against a partial clone of grafana/mimir at commit 4b45edce (October 27, 2026); no repository code was run. The population is every non-merge commit reachable from that commit that changes any of the five paths the runbook has had, as listed by git log --follow on its current path (two playbooks.md paths under jsonnet and operations, and three mimir-runbooks paths under docs/sources), each compared with its parent using rename detection. The full-history path query finds one commit that the --follow listing misses; both are counted from the path query. Removed and added lines are compared as multisets after trimming trailing spaces and dropping blank lines, so moved lines cancel.
A revision edits existing text when at least one removed line is not re-added in the same commit, and only adds text when it leaves no net removed line; the other 6 change no text once blank lines, trailing spaces and moved lines are ignored, and 2 of them are renames. The caution keyword filter matches whole words, case-insensitive: warning, caution, careful, carefully, danger, dangerous, do not, don't, never, data loss, irreversible and irreversibly. The incident filter matches incident, incidents, outage, outages, postmortem, post-mortem, on-call and oncall in the commit subject or body. One reader labeled the 48 keyword-flagged revisions as step-harm warnings or not, using the definition in the script's companion file, and labeled the 6 incident-word messages as naming a specific incident or not. The saved files hold commit hashes, dates, kinds, flags and labels, with no names.
The Amazon account is from its post-event summary, and the Knight account from the SEC order and press release, all seen through search excerpts on their own sites; the order's description of the removal is a paraphrase from the excerpt and is not quoted. Not measured: how often any runbook step caused harm in practice, Mimir's incident records, and executions of the runbook. The six-label rubric and the trace fields are editorial tools drawn up for this article, and the figure's series are illustrative.
Frequently asked questions
What is outdated runbook incident data?
It is a set of incident records in which a documented step ran as written and the impact grew because the system no longer matched an assumption in the procedure. Each record links the runbook version, step timestamps, the impact metric, any deviation and the runbook revision that followed.
How is this different from runbook exception data?
Exception data starts when a step fails and records what the engineer did next. Here nothing failed: the step succeeded, and the harm shows only when its timestamp is lined up against the impact metric and the system's change log.
How can you tell a harmful step from an incident that was getting worse anyway?
Compare the impact trend before and after the step, at a resolution finer than the gap between steps, and look for similar incidents where the step was run or skipped. If the rate did not change, label the case coincident worsening rather than blaming the procedure.
Do runbook revision histories explain why steps changed?
Rarely. In 356 revisions of Grafana Mimir's public runbook, two commit messages tie a change to a specific incident, and none of the nine revisions that warn about a step's side effects mentions one. The diff shows what changed; the incident tracker has to supply why.
What should a buyer require in a data request?
Ask for the runbook version in force, step start and end times with inputs, the impact metric sampled finely, the system change log, a deviation record and a revision link, plus one label from a published rubric. Ask for ordinary successful runs of the same runbook too, so harmful cases have a denominator.
Sources
- Amazon's summary of the February 28, 2017 S3 disruption: an authorized team member using an established playbook mistyped an input, removing more servers than intended; recovery times; the tool was changed to remove capacity more slowly with minimum-capacity safeguards. Source
- SEC order against Knight Capital (2013): after the August 1, 2012 problem was noticed, removing the new code from the seven correctly deployed servers worsened it, because incoming orders triggered old Power Peg code. Source
- SEC press release: Knight's router sent more than 4 million orders in the first 45 minutes while trying to fill 212 customer orders. Source
- Grafana Mimir repository at commit 4b45edce (October 27, 2026), from which SourceX counted 356 runbook revisions, 216 that rewrite existing text, 9 step-harm warnings and 2 commit messages tied to a specific incident. Source
- An August 2020 revision added a dry-run flag to a recovery upload command and warned against a flag that deletes blocks missing from the local filesystem. Source
- A May 2026 revision corrected guidance that treated a corrupted level-1 block as safe, because block-builder deployments do not replicate those blocks. Source
- A July 2026 revision moved the manual forget step last, added a scale-down check and warned that forgetting a live instance can lose data. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.