Systems and records
Security false positive outcome data: what a dismissal must record
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A dismissed security alert should carry a disposition from a fixed list that says why it was closed (detection logic wrong, data wrong, authorized test, expected activity, duplicate or insufficient evidence), the evidence that rules out malicious activity, and the follow-on record such as a tuning ticket. A bare false positive label hides which of these happened, and those causes teach a model opposite lessons.
Grader rubric
Security false positive
Key takeaways
- False positive and benign positive are different outcomes: one says the detection was wrong, the other says the activity was real but allowed.
- Split false positives by cause, detection logic or inaccurate data, because the fixes and the lessons differ.
- Every label needs minimum evidence an independent reviewer can check; a closing note that says fine is not evidence.
- Insufficient evidence and unreviewed automatic closure are honest labels and belong in the data.
- Keep reversals as a second dated decision instead of overwriting the original call with hindsight.
What reason and evidence should accompany a dismissed security alert?
A dismissed alert needs three things: a disposition from a fixed list that says why it was dismissed, the evidence that rules out the malicious explanation, and a pointer to what changed afterwards. Why matters more than the word false positive. A detection that fired on corrupted log data, a detection whose logic is wrong, and a detection that correctly caught an authorized penetration test are all closed without action, but they teach a model opposite lessons.
This guide gives security operations and managed security teams a disposition label guide they can apply to past alerts before those alerts become training or evaluation data. It covers the labels, the minimum evidence for each, a crosswalk to the values used by common SIEM and XDR tools, an illustrative worked example, the exception that breaks most label sets, and what to keep out of any dataset that leaves the company.
It deliberately does not cover how to investigate an alert. The question here is narrower: once a competent analyst has dismissed it, what must the record contain for someone else to trust that dismissal later?
Why is closed as false positive the weakest label in an alert history?
Because it hides whether the detection was wrong, the data was wrong, or the activity was real but allowed. NIST's glossary defines a false positive as benign activity or content incorrectly classified as malicious, which is a statement about the detection, not about the activity's harm.
The tools SOC teams already use make this split, but teams often collapse it at closing time. Microsoft Sentinel's incident API offers four classifications (true positive, benign positive, false positive and undetermined) and a separate classification reason that separates incorrect alert logic from inaccurate data and suspicious but expected activity. The Microsoft Graph security alert resource pairs a classification with a determination such as security testing, line-of-business application, confirmed activity, not malicious or not enough data to validate. Splunk Enterprise Security lists dispositions that distinguish false positives caused by incorrect analytic logic from those caused by inaccurate data, and benign positives that were suspicious but expected.
If your history only says closed or false positive, the distinction is lost and cannot be recovered from the label. It can sometimes be recovered from the notes and linked tickets, which is why the evidence fields below matter more than the label itself.
The disposition label guide: labels, minimum evidence and misuse
Apply one label per dismissed alert, chosen by the analyst who closed it or by a reviewer reconstructing the decision from the record. Each label has a minimum evidence requirement. An alert that cannot meet the minimum for any label gets insufficient evidence, which is itself a useful label, not a failure.
| Label | Use when | Minimum evidence attached | Do not use when |
|---|---|---|---|
| False positive: detection logic | The rule fired on activity that does not match the behavior it was written to catch | Rule identifier and version, the matching events, and why they fall outside the rule's intent | The rule did what it was written to do and the activity was authorized |
| False positive: inaccurate data | The alert came from wrong, duplicated, mis-parsed or mis-timed telemetry | The source record, the parsing or ingestion defect, and the corrected value | The data was accurate and the analyst simply judged the activity harmless |
| Benign positive: authorized testing | The activity was real and malicious-looking, and it was part of an approved test | The test authorization or scope document, the tester identity as a role, and matching time window and targets | The test scope does not cover the asset, account or time of the alert |
| Benign positive: expected administrative or business activity | A known user, administrator or internal application performed the activity for a legitimate reason | The change ticket, approved job or application inventory entry, plus the account and asset context | The only evidence is the user saying it was them |
| Duplicate | Another alert or incident already covers the same activity | The identifier of the surviving alert or incident | The two alerts share an indicator but describe different activity |
| Insufficient evidence | The analyst could not confirm or rule out malicious activity with the data available | What was checked, what was missing, and the retention or access limit that stopped the check | The analyst ran out of time but the data existed; record that as unreviewed instead |
| Unreviewed automatic closure | A suppression rule, playbook or age-out closed the alert without a human decision | The rule or playbook identifier and its version at closure time | A human reviewed and agreed with the automation; use the human's label |
What evidence explains the dismissal beyond a closed ticket?
The evidence that matters is whatever an independent reviewer would need to reach the same conclusion without asking the original analyst. A closing comment such as checked with user, fine is a conclusion with no support. Google Security Operations, for example, requires a close reason and a root cause in its Close Case dialog and lets administrators add their own root causes, which is a good prompt, but a root-cause value is still a summary until it links to evidence.
At a glance
What evidence explains the dismissal beyond a closed ticket
- Alert payload as raised: rule or model identifier, version, severity and the events that matched, with timestamps in one timezone.
- The checks performed: queries run, enrichment consulted, endpoints or accounts examined, recorded as references rather than pasted secrets.
- Authorization artifacts: the change ticket, test scope, job schedule or application inventory entry that explains the activity.
- People involved, as roles: closing analyst, reviewer, and the system owner or user who confirmed the activity.
- Runbook or playbook version in force when the alert was closed, because a later runbook may require checks the analyst was not asked to do.
- Follow-on records: a tuning ticket, suppression rule or parser fix created because of the alert, with its date.
- Reopen history: whether the alert or a related one was reopened later, and what the new evidence was.
How do tool-specific values map onto one disposition set?
Map each tool's closing values onto the guide's labels with a crosswalk, and keep the original value alongside the mapped one so nothing is lost if the mapping is later corrected. The table below uses values as published in each vendor's or project's own documentation. OCSF, the open cybersecurity schema, offers a normalized verdict list that includes false positive, benign, test, insufficient data, duplicate and managed externally, which makes it a convenient target field for the mapped value.
Where a tool has no value for a label, the gap is the finding: an analyst who could only choose false positive for an authorized test needed a better option, and a reviewer has to read the notes to recover the real reason. Google Security Operations' close reasons (malicious, not malicious, maintenance and inconclusive) are broader, so its root-cause field carries most of the detail.
| Guide label | Sentinel classification and reason | Graph security alert classification and determination | Splunk ES disposition | OCSF verdict |
|---|---|---|---|---|
| False positive: detection logic | FalsePositive, IncorrectAlertLogic | falsePositive, notMalicious | False Positive - Incorrect Analytic Logic | False Positive |
| False positive: inaccurate data | FalsePositive, InaccurateData | falsePositive, notMalicious (detail in comments) | False Positive - Inaccurate Data | False Positive |
| Benign positive: authorized testing | BenignPositive, SuspiciousButExpected | informationalExpectedActivity, securityTesting | Benign Positive - Suspicious But Expected | Test |
| Benign positive: expected activity | BenignPositive, SuspiciousButExpected | informationalExpectedActivity, confirmedActivity or lineOfBusinessApplication | Benign Positive - Suspicious But Expected | Benign |
| Duplicate | No dedicated value; record in the comment | No dedicated value; record in comments | No default value; a custom disposition can be added | Duplicate |
| Insufficient evidence | Undetermined | falsePositive, notEnoughDataToValidate | Undetermined | Insufficient Data |
How do you relabel a backlog of alerts that were closed with one generic value?
Relabel from the record, in batches grouped by detection rule, and let the evidence rather than the reviewer's memory decide. Grouping by rule works because a rule that fired on bad data or flawed logic usually did so many times, and one confirmed parser defect or rule fix can explain a whole batch once its dates are known.
For each rule, first collect the tuning tickets, parser fixes, rule version changes and test schedules from the same period. Then walk the closed alerts in date order and assign labels only where a linked record supports them; everything else becomes insufficient evidence. Have a second reviewer check a sample from every batch, and give any batch where the two reviewers disagree often a fresh look rather than settling it by majority vote.
Keep the original closing value in its own field and record who relabeled each alert, as a role, and when. A relabeled history is a reconstruction, and a buyer or auditor should be able to tell reconstructed labels from the ones analysts chose at closing time.
Worked example (illustrative): three dismissals from one week
The alerts, systems and details below are invented. A mid-sized company's SOC dismissed three alerts in one week. A reviewer preparing the history for a labeled dataset applies the guide and finds that only one of the three is usable as first recorded.
- Only the parser case was correct as originally labeled; the testing case was mislabeled and the travel case was unsupported.
- Relabeling did not need new investigation, only the records the team already had, which is the usual situation.
- The impossible-travel case is still valuable: an honest insufficient-evidence label teaches a model when not to conclude.
| Alert as closed | Evidence in the record | Label after review | Usable as is? |
|---|---|---|---|
| Credential dumping tool on a server, closed as false positive | Test scope document naming the server and window, tester role, matching timestamps | Benign positive: authorized testing | Yes, after relabeling from false positive |
| Impossible travel sign-in, closed as false positive | Note says VPN; no VPN log reference, no user confirmation | Insufficient evidence | Only as an insufficient-evidence example |
| Spike in failed logons, closed as false positive | Parser ticket showing a time-zone defect that duplicated events, with the fix date | False positive: inaccurate data | Yes |
| Follow-up for the logon spike | Parser fix deployed; no recurrence in the following month | Outcome recorded | Yes, adds the after state |
When a dismissed alert turns out to be real
Keep both decisions. When later evidence shows that a dismissed alert was part of a real intrusion, the original dismissal and the reversal are the most informative records in the history, as long as they are kept as two dated decisions rather than one overwritten field.
Record the original label, the evidence available at the time, the new evidence, the date it appeared and the corrected label. Do not retroactively mark the first analyst's call as an error if the evidence at the time supported it; mark it as superseded. A training set that silently rewrites past labels with hindsight teaches a model to expect information it will not have when it runs.
Reversals also usually involve the most sensitive material, so they are the cases most likely to be held back under the rules in the next section.
Which dismissed alerts should stay out of a dataset that leaves the company?
Anything that would let a reader attack you or your clients, and anything your contracts keep confidential, stays out. Dismissed alerts feel low risk because nothing bad happened, but their evidence often contains exactly the detail an attacker wants: what you monitor, what you ignore and why.
- Live secrets of any kind: passwords, tokens, keys, session cookies or connection strings found in alert payloads, which should be removed before export and treated as an incident if exposed.
- Detection logic that would reveal blind spots, such as exact suppression conditions or allow-listed paths, unless it is generalized.
- Client telemetry handled under managed service agreements, unless the client has agreed in writing to that use.
- Alerts tied to HR, legal or insider investigations, and any open investigation.
- Internal hostnames, IP plans and account names: replace them with consistent pseudonyms so relationships survive.
- Penetration test reports themselves, which belong to the testing agreement; keep only the scope facts the label needs.
Licensing a labeled dismissal history
A SOC or managed security provider that can show why each alert was closed, with the evidence linked, has records few public datasets offer. SourceX helps security providers turn selected dismissal records into licensed datasets for AI, handling rights review, contracting, preparation and delivery.
You choose which records are in scope. No alert record is delivered until the company has signed off on scope and terms in a signed agreement, and the starting point is a short company assessment with no upload of alert data.
Frequently asked questions
What is the difference between a false positive and a benign positive alert?
A false positive means the detection was wrong: the activity did not match what the rule was meant to catch, or the data was faulty. A benign positive means the detection was right about the activity but the activity was allowed, such as an authorized test or a known administrator task. Microsoft Sentinel and Splunk Enterprise Security both keep these as separate values.
What should an analyst write when closing an alert as a false positive?
Name the reason (detection logic or inaccurate data), link the events and the rule version, and describe the check that ruled out malicious activity. Add the tuning or parser ticket if one was raised. A note that only says fine or checked is not enough for anyone else to rely on.
Should alerts closed by automation be used as training labels?
Only with a label that says so. A suppression rule or playbook closure records the rule's judgment, not an analyst's, so mark it as an unreviewed automatic closure and include the rule version. Mixing them with human decisions makes a model learn the automation's blind spots.
How should an insufficient evidence dismissal be labeled?
Label it as insufficient evidence and record what was checked, what was missing and why. Sentinel calls this undetermined; OCSF has an insufficient data verdict. These cases teach a model when the right answer is that it cannot decide.
What if a dismissed alert later turns out to be malicious?
Keep the original decision and add the reversal as a second dated decision with the new evidence. Mark the first label as superseded rather than wrong if the evidence at the time supported it. Overwriting it hides what an analyst could actually know when the alert fired.
Sources
- NIST's glossary defines a false positive as benign activity or content incorrectly classified as malicious (SP 800-86 and SP 800-83 Rev. 1 definitions). Source
- The Microsoft Sentinel incident API (2025-06-01) classifies closed incidents as Undetermined, TruePositive, BenignPositive or FalsePositive, with reasons SuspiciousActivity, SuspiciousButExpected, IncorrectAlertLogic and InaccurateData. Source
- The Microsoft Graph security alert resource has a classification (falsePositive, truePositive, informationalExpectedActivity) and a determination such as securityTesting, confirmedActivity, lineOfBusinessApplication, notMalicious or notEnoughDataToValidate. Source
- Splunk Enterprise Security 8.5 lists dispositions Undetermined, True Positive - Suspicious Activity, Benign Positive - Suspicious But Expected, False Positive - Incorrect Analytic Logic and False Positive - Inaccurate Data. Source
- Splunk Enterprise Security dispositions default to Undetermined and administrators can configure custom dispositions. Source
- Google Security Operations requires a close reason and a root cause in the Close Case dialog, and administrators can add custom root causes. Source
- Google Security Operations case close reasons are Malicious, Not Malicious, Maintenance and Inconclusive. Source
- The OCSF dictionary's verdict_id lists normalized incident verdicts including False Positive, True Positive, Benign, Test, Insufficient Data, Duplicate and Managed Externally. Source
See if your company qualifies
A short company assessment. No data uploads are needed.