Skip to content

Privacy and preparation

Security alert triage disagreement data: keep every analyst's call

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Security training data should represent analyst disagreement as separate, timestamped assessments for the same alert, each with a verdict, a confidence value, the evidence cited and the runbook version, followed by an adjudication record with its rationale and remaining dissent, and a later outcome limited to what was verified. Collapsing the case to the final closing code throws away the reasoning that makes hard alerts hard.

License terms

Security alert triage

This page covers security alert triage disagreement data: keep every analyst's call.Illustrative

Key takeaways

  • Most SOC tools keep one verdict per incident, so disagreement survives only if you record it deliberately.
  • Store each analyst's assessment, the adjudication and a bounded outcome as separate, dated records.
  • Record the runbook version with each call so later reviewers judge analysts by the rules of the time.
  • Strip secrets, infrastructure detail and detection logic before export, and log every redaction.
  • Keep open investigations, legal holds and contested adjudications out of datasets and evaluation scores.

How should security training data represent analyst disagreement about an alert?

Represent each analyst's assessment as its own record, keep the adjudication as a separate record with its rationale, and attach a later, bounded outcome that says what was actually confirmed. Never collapse the case to the final closing code. A dataset that only stores 'benign positive' cannot teach a model why a competent analyst once called the same alert a true positive, and that reasoning is often the most useful signal in the case.

In practice, each assessment needs the analyst's role, the verdict, a confidence value, the evidence they cited, the runbook version they followed and the time they made the call. The adjudication record names the reviewer's role, the evidence that settled it and the dissent that remains. The outcome record is dated after both and limited to what was verified, such as 'no further activity from this account in 30 days' rather than 'confirmed safe'.

This page is for security operations and IT service leaders deciding how to keep disputed triage decisions so that they can later support model training, evaluation or a licensing review. It assumes alerts are already handled in a SIEM, a case tool or a ticketing system, and it covers what to add on top.

Why does the closing classification erase the argument?

Most security tooling stores one verdict per incident, so a disagreement survives only if someone writes it into a comment. The Open Cybersecurity Schema Framework, for example, gives its Incident Finding class a single verdict_id with values such as False Positive, True Positive, Benign, Suspicious, Insufficient Data and Duplicate, alongside a lifecycle status and an assignee. That design is right for operations, where an incident needs one current answer.

Microsoft Sentinel's incident API works the same way. Its classification field takes one of Undetermined, TruePositive, BenignPositive or FalsePositive, with a reason such as SuspiciousActivity, SuspiciousButExpected, IncorrectAlertLogic or InaccurateData. Two analysts who disagreed leave one classification behind, and the losing view lives, at best, in free-text notes that are hard to align with the evidence.

For training data, that single value is a lossy summary. It hides how hard the case was, which evidence was ambiguous and whether the final answer depended on context that arrived later. A model trained only on closing codes learns that hard cases are easy.

Which kinds of analyst disagreement are worth keeping?

Not every disagreement deserves a record. Keep the ones where two qualified analysts, reading the same evidence under the same runbook, reached different conclusions that would change what happens next. Drop differences in wording and cases where one analyst simply had not opened the evidence yet.

The table separates the common types, because each one needs a different label policy.

Which kinds of analyst disagreement are worth keeping?
Disagreement typeIllustrative formWhat to preserveLabel consequence
VerdictTrue positive versus benign positiveBoth verdicts, evidence each cited, adjudicated verdictKeep as a soft or multi-label target, not a single class
Cause of a false alarmFaulty detection logic versus bad input dataRule version, data source health at alert timeRoute to detection engineering labels, not triage labels
Severity or priorityMedium versus high for the same confirmed activitySeverity rubric version and the factors each analyst weighedTrain severity separately from the verdict
ScopeOne host affected versus severalEntity lists each analyst attached, later scoping evidenceUse the outcome record, not either analyst, as the scope label
Evidence sufficiencyClose now versus wait for more telemetryTime of each call and what arrived afterwardsLabel as insufficient at the time; never back-fill hindsight
Escalation pathHandle in the SOC versus escalate to incident responseEscalation criteria in the runbook version usedKeep as a process decision, separate from the verdict

Which security standards already record more than one assessment?

Threat intelligence formats already treat assessment as something several parties can hold at once, and they make a good pattern for triage data. In STIX 2.1, an Opinion object records an assessment of another STIX object, produced by a different entity, on a fixed five-point scale from strongly-disagree to strongly-agree, with the authors, an explanation and the objects it refers to. Common properties in the same schema include a confidence integer from 0 to 100 that expresses the creator's confidence in the correctness of their data.

You do not need to adopt STIX to use the idea. Model each analyst's view as an assessment record that points at the alert, has an author role and an explanation, and carries a confidence value on a documented scale. The adjudication becomes one more assessment with authority to set the operational verdict, while the others stay in the record.

Keep the confidence scale explicit. OCSF's confidence_id, for example, describes the accuracy of the analytics that produced a finding, which is a property of the detection, not of the analyst. Mixing detection confidence and analyst confidence in one field is a common way to make a dataset unreadable.

Which fields should a disputed-alert dataset brief require?

The asset on this page is a dataset brief that a SOC lead and a domain reviewer can fill in together before any records are extracted. Each row is a field the dataset must carry, the evidence that fills it and the check that rejects a case. A case that fails any check stays out of the training split and goes to a review queue.

At a glance

Which fields should a disputed-alert dataset brief require

The brief is written for disputed cases, but the same fields work for agreed ones. Including a matched sample of agreed cases lets you measure whether a model can tell easy alerts from contested ones.

Which fields should a disputed-alert dataset brief require?
FieldFilled fromRejection checkReviewer
case_refPseudonymous incident and alert IDs mapped to the source system in a key kept inside the SOCAny live hostname, username or ticket URL in the exported valueSecurity reviewer
chronologyAlert time, each assessment time, adjudication time, outcome timeAny assessment without a timestamp, or out of orderDomain reviewer
access_and_approvalWho approved export of this case and under which policyNo named approving role or policy referenceSOC lead
alert_evidenceReferences to the telemetry each analyst viewed, with redacted excerptsRaw secrets, tokens, keys or full command lines with credentialsSecurity reviewer
runbook_versionVersion or date of the triage procedure in forceMissing, or newer than the alertDomain reviewer
assessmentsOne entry per analyst: role, verdict, confidence on a stated scale, evidence cited, explanationFewer than two qualifying assessments for a disputed caseDomain reviewer
adjudicationReviewer role, final verdict, evidence that decided it, remaining dissentNo rationale, or rationale citing evidence not in alert_evidenceDomain reviewer
bounded_outcomeWhat was later verified, the window and the sourceClaims beyond what the outcome source showsDomain reviewer
redaction_logWhat was removed or generalized and by which ruleEmpty log for a case containing free-text notesSecurity reviewer
sharing_labelInternal handling label and any client or contract restrictionRestriction that forbids external useSOC lead and counsel

Where does the reasoning behind a triage call actually live?

A closed ticket shows the last state; the explanation lives in the sequence of evidence and decisions around it. Rebuild that sequence from the case tool's history, the SIEM queries saved with the case and the notes each analyst left, then check it against the runbook in force on the day.

The runbook version matters more than teams expect. If the procedure changed a week after the alert, a reviewer reading today's version may judge an analyst wrong who followed the rules of the time. Store the version, or at least its effective date, with every assessment.

Bound the outcome to what was verified. NIST's incident response guidance, Special Publication 800-61 Revision 3, superseded Revision 2 in 2025 and organizes incident response around the CSF 2.0 Functions, including continuous improvement from lessons learned. A lessons-learned review is a good outcome source when it exists, because it is written after the facts settle. Where none exists, a narrow observation (no recurrence from the same entity within a stated window) is better than an unsupported claim of innocence or compromise.

Worked example (illustrative): an impossible-travel sign-in

This example is illustrative; it does not describe a real organization, user or incident. An identity alert fires for a sign-in from two distant locations within an hour for one employee account.

  • Assessment 1, tier 1 analyst: true positive, confidence 70 on a 0 to 100 scale. Evidence cited: the second sign-in came from an IP address not seen for this user and used a new device. Explanation: credential theft pattern under the current runbook.
  • Assessment 2, tier 2 analyst: benign positive, confidence 60. Evidence cited: the second IP sits in a range the analyst recognized as a newly added corporate VPN egress point. Explanation: expected activity; the detection's allow list is out of date.
  • Adjudication, SOC lead: benign positive, with cause recorded as detection logic, because the network team's change record showed the VPN range was added three days before the alert and not yet in the allow list. Dissent kept: assessment 1 was reasonable under the evidence and runbook available to tier 1.
  • Bounded outcome: no further anomalous sign-ins for the account within 30 days, per the identity provider logs; the detection rule was updated within the week. The outcome claims nothing about the device beyond that window.
  • Exception to watch: if a later investigation had found the VPN range itself was misused, the adjudicated verdict would be wrong. The dataset keeps the original adjudication and adds a dated correction rather than overwriting it, so the case teaches both the decision and its revision.

Which secrets and detection details must be stripped from triage history?

Triage records are full of material that is dangerous outside the SOC: credentials pasted into notes, internal hostnames and IP plans, the exact logic of detections and the gaps between them. Strip or generalize these before export, and log each change in the redaction log so reviewers can see what the model will never see.

Sharing labels help but do not settle the question. FIRST's Traffic Light Protocol defines four labels, TLP:RED, TLP:AMBER, TLP:GREEN and TLP:CLEAR, that indicate the sharing boundaries recipients should apply, and FIRST says TLP was not designed to handle licensing terms. A TLP:CLEAR label on a note does not make it licensable, and a TLP:RED label is a strong reason to keep the case out entirely.

Detection content needs its own rule. Generalize rule names to their technique or behavior, remove thresholds that would let a reader evade the detection, and never export live allow lists. The adjudication rationale can still say 'the allow list was out of date' without revealing what was on it.

Which disputed alerts should stay out of the dataset for now?

Some disputes should never enter a dataset, and others should wait. Exclude cases tied to an open investigation, a legal hold, a regulatory notification or a law enforcement request; their facts are not settled and their handling is governed by people outside the SOC.

Managed security providers face an extra limit: the telemetry and notes usually concern a client's environment, and client agreements commonly treat that material as confidential. The provider's own process records may be reusable where the client's data is not, and counsel should read the agreements before any case leaves the building.

Bring in a qualified reviewer whenever adjudication itself is contested, for example when the SOC lead and incident response disagree about whether an event was an incident. Mark those cases unresolved and keep them out of evaluation scoring; a benchmark built on contested answers measures the label, not the model.

Can a SOC license its adjudicated triage history?

Adjudicated alert histories are the kind of operational record SourceX helps a security operation license to AI developers, with SourceX running the rights review, contract, preparation and delivery. The SOC's company picks the cases and fields in scope and signs before any case leaves, and afterwards the de-identified set is used only as that signed agreement permits.

Whether a security team's records could qualify is checked first through a brief company questionnaire; no telemetry or notes are uploaded for it.

Frequently asked questions

Should disputed alerts be removed from security training data?

Usually not. Disputed cases carry the most information about where judgment is hard, so keep them with every assessment and the adjudication. Remove them only from evaluation splits where you need one settled answer, or when the adjudication itself is still contested.

How many analysts need to review a case for it to count as disputed?

Two qualifying assessments that reach different conclusions under the same runbook are enough. More reviewers help measure agreement rates, but the brief's requirement is that each assessment is independent, timestamped and backed by cited evidence.

Is the final classification in the SIEM the ground truth?

It is the operational answer at closing time, which is not the same thing. Treat it as the adjudicated verdict and look for a later, bounded outcome record, such as a lessons-learned review or a recurrence check, before calling anything ground truth.

Can analyst notes be used for fine-tuning?

Only after redaction of secrets, internal infrastructure detail and personal data, and only where the organization has the right to use them that way. Notes from a managed provider about a client's systems usually need the client's permission, whatever the redaction quality.

What confidence scale should analysts use?

Any documented scale works if everyone uses the same one and it is stored with the record. A 0 to 100 integer like the STIX confidence property is easy to aggregate; a short ordinal scale is easier for analysts to apply consistently.

Sources

  • OCSF defines verdict_id values including False Positive, True Positive, Benign, Suspicious, Insufficient Data and Duplicate, and confidence_id as the accuracy of the analytics that produced a finding. Source
  • The OCSF Incident Finding class carries one verdict_id, a lifecycle status_id and an assignee for each incident. Source
  • The Microsoft Sentinel incident API takes one classification (Undetermined, TruePositive, BenignPositive or FalsePositive) with reasons such as SuspiciousActivity, SuspiciousButExpected, IncorrectAlertLogic and InaccurateData. Source
  • The STIX 2.1 Opinion object records another party's assessment on a five-point scale from strongly-disagree to strongly-agree, with authors, an explanation and object references. Source
  • STIX common properties include a confidence integer from 0 to 100 for the creator's confidence in the correctness of their data. Source
  • FIRST's Traffic Light Protocol 2.0 uses four labels to indicate sharing boundaries and was not designed to handle licensing terms. Source
  • NIST SP 800-61 Revision 3 (2025) supersedes Revision 2 and organizes incident response recommendations around the CSF 2.0 Functions, including lessons learned. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies