AI uses for records
Alert response decision dataset: the false positives that were real
By SourceX Editorial · Updated
9 sections
Short answer
An alert response decision dataset records three separate verdicts for every firing: whether the condition was real, whether it mattered then, and whether anyone needed to act. Wikimedia's public alert rules show why the split matters: of 83 tuning commits that explain unwanted firings, 68 describe real conditions that needed no response, and only 2 describe a broken measurement.
Agent run
Alert response decision
Key takeaways
- Detection and response are separate verdicts: a rule can read the system correctly and still interrupt someone for nothing.
- In Wikimedia's alert rules history, 68 of 83 commits that explain unwanted firings describe real conditions that needed no response; 2 describe a broken measurement.
- 27 of 34 commits that say false positive describe conditions that really occurred, so the phrase cannot be copied as a label.
- The unnecessary verdict arrives late: a median of 78 days passed between an alert's creation and its first quieting change.
- A usable record joins seven pieces: firing, impact, acknowledgment, action, suppression, later tuning and ownership at firing time.
What is an alert response decision dataset?
An alert response decision dataset is a set of alert firings in which each firing carries three verdicts kept apart: whether the measured condition really happened, whether it mattered at that moment, and whether a person needed to act. Most alert exports collapse those into one closing status, and that status cannot tell a broken detector from a working detector attached to the wrong urgency.
The distinction matters because the two failures call for opposite fixes. A detector that misreads the system needs a better query or a repaired exporter. A detector that reads the system correctly but wakes someone for a condition that clears by itself, sits inside the service level objective or belongs to another team needs a different hold time, severity or route. A model trained on one merged label learns neither repair.
This article measures how often the second case appears in a public alerting history, shows which records separate the verdicts, and gives a rubric a buyer can apply firing by firing. The evidence is Wikimedia's production alert rules repository, chosen because its rules, its severity levels and its full change history are public and versioned.
Why does an accurate alert still end in an unnecessary response?
An accurate alert ends in an unnecessary response when the rule correctly detects a condition whose cost does not justify interrupting anyone. Alerting guidance treats this as a known trade-off rather than an accident. The Prometheus project's alerting practices summarize the goal as keeping alerting simple, alerting on symptoms and to "avoid having pages where there is nothing to do".
Google's SRE workbook chapter on alerting on SLOs names the measure directly: it defines precision as the share of detected events that were significant, so a rule can fire on real conditions and still have low precision. Detection and significance are two different properties of the same firing.
Response processes add a second, deliberate source of over-response. PagerDuty's public incident response guide tells responders who are unsure between two severities to "treat it as the higher one" and to "just assume the highest and review during a postmortem". The severity recorded during the incident is therefore provisional by design, and a dataset that stores only that first value stores the cautious guess, not the reviewed verdict.
Tooling also hides part of the picture before any human sees it. Alertmanager's configuration reference states that "If an alert is resolved before group_wait has elapsed, no notification will be sent for that alert." A firing that cleared quickly may exist in the rule engine and be absent from the paging tool, so the denominator of firings depends on which system a dataset was exported from.
What 99 Wikimedia tuning commits say about real alerts nobody needed
Wikimedia's alert rules repository shows that when engineers explain why they quieted an alert, they nearly always describe a real condition that did not need a response. The repository holds the Prometheus alert rules Wikimedia deploys to production, one folder per team, and its test suite, run at code review, rejects a rule without a severity label or without summary, description, dashboard and runbook annotations (a TODO placeholder is accepted).
At commit 40e0f79, the state on October 27, 2026, 989 non-merge commits change at least one rule file, starting June 1, 2021. Of those, 428 change an existing alert's severity, hold time or expression, or remove an alert. Changes that only quiet alerts outnumber changes that only make them louder by 124 commits to 33. At the level of single alerts, hold times were lengthened 71 times and shortened 13 times.
Most tuning commits give no reason in words. A whole-word search of the commit messages for terms such as noisy, flapping, false positive, actionable, maintenance, depooled, transient and page finds a reason word in 99 of the 428 tuning commits (23.1 percent). One reader then labeled those 99 by what each message says about the firings it changes. Sixteen turn out not to be about unwanted firings at all (they add paging, reroute, refactor or revert), which leaves 83 that discuss whether past firings deserved a response.
Of those 83, 68 (81.9 percent) describe a condition that really occurred but did not need anyone to act, 13 say an alert was noisy or a false positive without saying why, and only 2 (2.4 percent) describe a measurement that misreported the system. One of the two is an alert that flapped because OpenStack metrics were scraped every 15 minutes while Prometheus returns only the last 5 minutes by default. The table splits the 68 real-but-unneeded cases by the reason the message gives:
| Reason the response was unneeded | Commits | What a message says (Wikimedia, paraphrased unless quoted) | Record that would confirm it |
|---|---|---|---|
| Expected during planned or routine work | 16 | "Neutron failing for cloudvirt in maintenance is fine and expected" (2026) | Maintenance window, depool or deploy log for the same host and time |
| Outside the alert's purpose | 16 | Staging certificates built to last a couple of days set off a daily expiry alert (2025) | The rule's stated scope next to the labels of the firing series |
| Cleared on its own | 15 | Multiple pages for events that auto-recovered, so the window grew to 5 minutes (2023) | Firing and resolve timestamps; resolve before acknowledgment |
| Below the impact that needs a response | 15 | Non-critical services should open a ticket rather than page (2026) | SLO burn or user-facing error rate at firing time |
| Covered by another alert | 6 | Paged with no user impact while blackbox checks already caught real impact (2024) | The other alert's firing history over the same window |
Why the words false positive cannot be a training label
The phrase false positive cannot serve as a label because the engineers who write it usually mean real but unneeded. Thirty-four of the 99 reason-word commits use false positive, false positives, false alarm or false alarms. By the same reading, 27 of those 34 (79.4 percent) describe conditions that genuinely occurred: hosts depooled for maintenance, staging certificates that really were about to expire, backup database replicas that really were lagging, latency on a data center that had been drained of traffic.
Only one of the 34 describes a detector that reported something that was not there: a traffic-drop alert fed by a log-tailing exporter with a known bug, which the team moved to a more reliable metric in 2021. The remaining six either give no cause or use the phrase in another sense.
For a buyer this changes how closure codes are read. An alert or incident closed with a false positive note, in whatever paging or ticketing tool, may mean the detector was wrong, the alert fired during approved work, or the threshold was set below what anyone cares about. Those teach opposite lessons to a triage model, so the closure code has to be re-derived from the records described below, not copied.
How long until an alert is judged unnecessary?
The verdict that a response was unnecessary often arrives weeks or months after the firing, so a dataset cut too early labels those firings wrongly. Following each alert from the commit that created it, 74 of the 676 alerts created in the repository were later demoted, lost paging or had their hold time lengthened (10.9 percent). The median time from creation to that first quieting change was 78 days; 15 came within a week, 27 within 30 days and 55 within a year. Separately, 95 alerts were later removed.
Paging alerts are not exempt. Of the 35 alerts created with severity page, 7 were later quieted and 5 removed. The best single record in the repository shows what a deliberate review looks like: before lengthening one alert's hold time from 30 to 45 minutes in February 2026, an engineer wrote that "Analysis of alert firings from Jan 21 - Feb 4 showed: - 54 incidents total - 41% resolved within 15 minutes of firing". That message joins firing timestamps, a resolution pattern and a rule change in one place, which is exactly the join a dataset needs.
Two practical rules follow. Freeze the label window only after the rule's next review, or record the rule version in force at each firing and accept that later tuning is a retrospective verdict. And keep the firing even after its rule is removed, because a deleted alert is often the clearest case of a response nobody needed.
The alert response trace: seven records and the verdict each settles
Seven records separate a correct alert from a needed response, and each answers a different question. A dataset that lacks one of them should carry an explicit undetermined value for that question instead of inheriting the closing status.
The trace below lists them in the order a firing produces them. The outcome rubric that follows turns them into one verdict per firing:
- Firing record: rule name, rule version or commit, severity and team labels, the series that crossed the threshold and its values. It settles whether the condition occurred as measured.
- Impact evidence: SLO burn, user-facing error rate or latency for the affected service in the same minutes. It settles whether the condition mattered then.
- Delivery and acknowledgment: which receiver got it (page, chat, ticket), who acknowledged it and when, or that it resolved before anyone did. It settles whether a person engaged.
- Action log: runbook steps run, changes made, escalations, or an explicit no-action note. It settles what the response actually was.
- Suppression records: silences, inhibitions, maintenance windows and depool events overlapping the firing. They settle whether the state was expected.
- Later tuning: the rule change that cites the firing or its task, with its reason. It records the team's retrospective verdict.
- Ownership at firing time: the route and team the alert used, and any later move to another team. It settles whether the right people were asked to act.
| Verdict | Rule for assigning it | Training use | Common mislabel |
|---|---|---|---|
| Needed and taken | Condition confirmed, impact present, an action changed the outcome | Positive example for paging and escalation | Closed as resolved with no action recorded |
| Needed and missed | Condition and impact confirmed, no acknowledgment or action before harm | Recall failures for detection and routing | Absent entirely when the paging tool never notified |
| Real, response unneeded | Condition confirmed; cleared on its own, expected, out of scope, covered elsewhere or below impact | Hold-time, scope and severity recommendations | Closed as false positive |
| Detection wrong | Firing series contradicted by raw data, exporter bug or query error | Detector repair, not routing | Folded into noisy |
| Undetermined | Any of the first four records missing for the firing | Exclude from gold sets; use for coverage audits | Guessed from the closing status |
The strongest objection: tuning commits only show alerts someone fixed
The fair objection is that a rules repository shows only the alerts someone cared enough to change, so it cannot measure how often firings are unnecessary overall. That is correct, and the counts above are not a rate of unnecessary pages. They measure what engineers wrote when they changed a rule, which is the retrospective half of the trace.
Three limits follow. Reasons appear in fewer than a quarter of tuning commits, so the labeled set is the explained minority. Acknowledgments, actions and impact live in the paging tool, chat and dashboards, none of which this repository holds. And one reader assigned the labels, so borderline calls such as outside the alert's purpose versus below impact should be checked by a second reader against the messages, which the methods note identifies.
The link that would close the gap already exists in part: 269 of the 428 tuning commits (62.9 percent) carry a Bug: trailer naming a Phabricator task, where firing discussions often live. A company archive with the same habit, plus its paging export, can rebuild the whole trace for most firings rather than the explained minority.
How buyers would use alert response records, and what to ask for
Buyers would use alert response records to teach and test the judgment between notify and wait, not the detection itself. A triage agent can be scored on whether it recommends paging, ticketing or no action for a firing whose verdict was later settled. An alert-tuning agent can be given a rule and its firing history and scored against the hold time, scope or severity the team eventually chose. Both need the verdicts kept separate, or the model is rewarded for silencing detectors that were right.
Ask for evidence before volume. A useful acceptance test checks, for a sample of firings, that the rule version at firing time is known, that acknowledgment and resolve timestamps exist, that suppression records cover the same hosts, and that later rule changes can be joined to the firings they cite. SourceX's PagerDuty and Opsgenie export guides describe where the acknowledgment half lives; this page covers how to label it. Security alert dismissals have their own disposition problem, covered in the false positive outcome article, and are best kept as a separate slice.
SourceX licenses business data directly from companies, and it sources datasets when a buyer asks rather than keeping them in stock. A request for alert response records works best when it states the intended use, the verdict labels required, the minimum share of firings that must carry each of the seven records, and the acceptance test above.
Methods note: the Wikimedia rules clone, labels and limits
Counts come from a script that reads git objects and YAML rule files from a clone of github.com/wikimedia/operations-alerts at commit 40e0f79 (committed October 27, 2026); no repository code was run. Every commit listed by git rev-list from that commit is read, which includes commits that arrived through merges. The 221 merge commits are skipped and every non-merge commit is compared with its single parent (the root commit with an empty tree), using rename detection. Rule files are paths starting team- and ending .yaml, excluding _test.yaml files.
An alert is identified by file and alert name; a name that leaves one file and appears in another within one commit counts as a move (45 cases), not a removal and an addition. Severity rank is page 4, critical 3, warning 2, task and ticket 1, info 0; a demotion lowers the highest rank in the alert's group. Hold time is the largest for duration in the group, parsed exactly. Expression changes compare whitespace-normalized text. A tuning commit changes an existing alert's severity, hold time or expression, or removes an alert. A quiet-only commit demotes, lengthens a hold, removes paging or removes an alert and does none of the opposite (promote, add paging, shorten a hold); a louder-only commit does one of those three and none of the four quieting changes. Each alert is followed from the commit that added it, through moves and the 17 renames of its rule file; its first quieting change is the first later demotion, paging removal or longer hold.
Reason words are matched whole-word and case-insensitive in the message with trailer lines removed: false positive(s), false alarm(s), noise, noisy, flap, flaps, flapping, flappy, spam, spammy, spurious, actionable, non-actionable, unactionable, page, pages, paging, non-paging, silence, silenced, maintenance, depooled, transient, too sensitive and fatigue. One reader labeled the 99 matching commits with eight labels and a fixed precedence (measurement wrong, planned work, cleared on its own, outside purpose, covered elsewhere, below impact, cause not stated, not about unneeded firings); the label file and definitions are kept with the script. A Bug: trailer is any message line beginning Bug:. Medians use exact elapsed seconds; percentages are rounded once, half up. Saved outputs hold commit hashes, dates, counts and labels, with no names.
Frequently asked questions
What is the difference between a false positive alert and an unnecessary alert?
A false positive alert reports a condition that did not occur, usually because of a broken metric, exporter or query. An unnecessary alert reports a condition that did occur but did not need a person: it cleared by itself, happened during planned work or stayed below the impact that matters. In Wikimedia's public alert history, 27 of 34 commits that use the phrase false positive or false alarm describe the second kind.
Which records prove that an alert needed a response?
The firing record shows the condition occurred, impact evidence such as SLO burn shows it mattered, and the acknowledgment and action log show that someone engaged and that the action changed the outcome. Without the impact and action records, a firing can only be labeled undetermined, however confident the closing status looks.
Can alert tuning history be used as training data on its own?
Only for the retrospective half of the decision. Rule changes with stated reasons show which firings a team later judged unnecessary and why, which is useful for training hold-time and scope recommendations. They do not show acknowledgments, actions or impact, so they cannot label whether a specific firing was answered correctly.
How long should a team wait before labeling alerts as noise?
Long enough for the rule's next review. In the Wikimedia repository the median time from an alert's creation to its first quieting change was 78 days, with 55 of 74 such changes inside a year. A label window shorter than the review cycle records the provisional judgment, not the settled one.
Do paging tools record firings that never notified anyone?
Not always. Alertmanager documents that an alert resolved before its group_wait has elapsed sends no notification, so a short-lived firing can exist in the rule engine and be missing from the paging tool. A dataset should state which system its firings were exported from and whether quick resolutions are included.
Sources
- Wikimedia's production Prometheus alert rules repository at commit 40e0f79: the full commit history counted in this article (989 non-merge commits changing rule files, 428 tuning commits, 99 with reason words). Source
- The repository's test file requires a severity label, summary, description, dashboard and runbook annotations, and a #page tag in the summary whenever severity is page. Source
- A February 2026 commit lengthening an alert's hold time from 30 to 45 minutes after an analysis of its firings from January 21 to February 4. Source
- An October 2026 commit making alerts for non-critical services open tickets rather than page because they should not wake people up. Source
- A July 2023 commit widening an alert window to 5 minutes after multiple pages for events that auto-recovered. Source
- A June 2022 commit explaining that an alert flapped because OpenStack metrics were scraped every 15 minutes while Prometheus returns only the last 5 minutes by default. Source
- An August 2026 commit gating an alert on host maintenance status because the failure during maintenance is fine and expected. Source
- Wikimedia's Alertmanager guide describes notification routing by alert severity, including task-level alerts that open Phabricator tasks (paraphrased from a search excerpt). Source
- Prometheus alerting practices: keep alerting simple, alert on symptoms, and avoid having pages where there is nothing to do. Source
- Alertmanager configuration reference: if an alert is resolved before group_wait has elapsed, no notification is sent for that alert. Source
- PagerDuty's incident response guide: when unsure between two severities, treat the incident as the higher one and review during a postmortem. Source
- Google SRE workbook, Alerting on SLOs: precision is defined as the proportion of detected events that were significant (paraphrased from a search excerpt). Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.