Skip to content

Privacy and preparation

Security operations dataset time splits across policy changes

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Split security operations records by the policy version in force, not at random. Cut the timeline at each change to a detection rule, runbook or approval policy, train on cases decided before it and test on cases decided after it, with a gap so open incidents do not straddle the cut. Then confirm every action was taken by someone authorized on that date.

Evaluation split

Security operations dataset

This page covers security operations dataset time splits across policy changes.Illustrative

Key takeaways

  • Random splits leak incidents, alert bursts and runbook versions across the boundary, so they cannot test whether reasoning adapts to a policy change.
  • Cut at the effective time of changes that alter the expected action, and quarantine incidents open across the cut plus a gap based on your own case durations.
  • Sigma's modified field keeps only the last change date, so rule versions in force must come from repository or SIEM history.
  • Every action needs a dated authority check against role history, approvals and the policy version, because a closed ticket proves only a status change.
  • Cases that fail the authority check stay in the dataset as labeled exceptions rather than answer keys.
  • Redact secrets, live detection logic and internal network details by replacing values, so the decision evidence survives.

How do you evaluate security workflow reasoning across policy changes?

Evaluate it on incidents that were decided after a policy change the model never saw in training. Order every case by when it was decided, cut the timeline at the effective date of each detection rule, runbook or approval policy change, train on the earlier side and test on the later side, and score whether the model follows the version that was in force for each test case.

A random split cannot measure this. Alerts from one incident, near-duplicate alerts from one noisy rule, and cases handled under the same runbook revision land on both sides, so a model can score well by recalling the old procedure. scikit-learn's own documentation for its time-series splitter says ordinary cross-validation is inappropriate for time-ordered data because it trains on the future and evaluates on the past.

The second half of the test is authority. A security record shows what someone did, not whether they were allowed to do it. A case where an analyst isolated a production host without the approval the policy required is a poor answer key even if the outcome looked right, so each action needs an authority check against the roles and approvals that applied on that date.

Which changes count as a policy change in a security operations record?

Any change that would make a correct decision on one date wrong on another. In practice four kinds of change matter, and each one keeps its version in a different place, so the first job is to find where each version history lives before anyone chooses cut points.

Detection logic is the most visible. The Sigma rule specification gives rules a creation date and a modified date, a status such as experimental, test, stable or deprecated, and a related field that marks a rule as derived from, merged into or made obsolete by another. The modified field holds only the last modification date, so it cannot tell you which logic was live in a given month; the repository or SIEM change history has to. OpenTelemetry's semantic conventions also define a security_rule.version attribute for the rule revision used in an analysis, which is the kind of field to keep on every alert.

Procedures change because they are meant to. NIST SP 800-53 control IR-4 asks organizations to feed lessons learned from incident handling back into procedures, training and testing. NIST's incident response guidance, revised in April 2025 as SP 800-61 Revision 3, deliberately leaves out step-by-step procedures because those details change too often and vary too much between organizations, so a team's own versioned runbooks are the authority for what was expected at the time.

Which changes count as a policy change in a security operations record?
Change typeWhere the version usually livesField to keep on each caseEffect on the split
Detection rule logic or thresholdRule repository history or SIEM rule audit logRule id, rule revision, revision effective timeCut at each material revision of rules that drive many cases
Rule retired or replacedRule status and related-rule linksSuperseding rule id, retirement timeAlerts from the retired rule stay on the training side
Runbook or playbook revisionWiki page history, SOAR playbook versionsRunbook id and revision cited by the caseTest only on cases opened after the revision went live
Approval or escalation policyPolicy document versions, change ticketsPolicy version, approver role requiredCut at the effective date; validate authority per version
Role or group membershipIdentity provider audit log, access reviewsActor role at action time, source of that factNo cut, but feeds the authority check on every action
Tooling or data source changeChange management records, connector configsTelemetry source and parser versionRecord as a covariate; cut only if fields or meaning changed

Where should the cut points go, and how wide should the gap be?

Put each cut at the effective time of a change that alters the expected action, and leave a gap after it at least as long as the time most incidents stay open. Changes that do not alter the expected action, such as a reworded runbook step, do not justify a cut and only shrink the test set.

The gap matters because security cases are not points in time. An incident opened under the old policy and closed under the new one can carry evidence from both sides, and a model trained right up to the cut may have seen the beginning of a case it is later tested on. scikit-learn's splitter has a gap parameter for the same reason: it drops samples from the end of each training window before the test window starts. Choose the gap from your own case durations, for example the time within which most incidents in the archive were closed, not from a default.

  • List candidate changes with effective timestamps, not approval dates; a policy approved on one day and switched on a week later is cut on the second date.
  • Keep a change as a cut only if a reviewer can state, in one sentence, an action that was correct before and wrong after.
  • Prefer one strong cut with enough test cases over several weak cuts that leave each test window too thin to compare.
  • Assign every case to a side by the time of the decision being scored, and quarantine cases whose open period crosses the cut plus the gap.
  • Keep a later window after the newest change untouched as a final holdout, so repeated tuning on the first test window does not leak into the reported score.

The time-scoped split and authority validation protocol

The protocol below is the asset: a step list a security operations lead and a data lead can run together on an exported case archive. Each step names its input, the rule applied and what happens when the check fails. Run it on a sample before the full archive so problems with timestamps or missing version history surface early.

The time-scoped split and authority validation protocol
StepInputRule appliedCheck or outputIf it fails
1. Freeze the case setCase export with ids and timestampsOne row per case; alerts nested under itCase count and date range recordedDeduplicate merged incidents before going on
2. Build the change logRule, runbook, policy and role historiesEvery change has an effective timestamp and authorVersioned change tableChanges without a timestamp are excluded as cuts
3. Choose cut pointsChange table and case volumes per windowExpected action differs before and afterCut list with one-line rationale eachMerge or drop cuts with too few later cases
4. Assign sidesCases, cuts, gap lengthDecision time decides the sideTrain, test, final holdout and quarantine labelsCases missing a decision time go to quarantine
5. Attach versionsEach case and change tableRule revision, runbook revision and policy version in forceVersion fields filled on every caseUnknown versions stay unknown, never inferred
6. Validate authorityActions, approvals, role historyActor held the required role or approval at action timePass, fail or unverifiable per actionFailed cases are kept as labeled exceptions, not answer keys
7. Check leakageTrain and test sidesNo shared incident ids, hosts or near-duplicate alert bursts across the cutOverlap reportMove overlapping clusters to quarantine
8. Screen for secretsCase text and attachmentsNo credentials, keys, internal IPs or exploit stepsRedaction log per caseDrop the case if redaction removes the decision evidence
9. Score by versionModel outputs on test casesAnswer judged against the version in force for that caseAccuracy before and after each cut, with intervalReport cuts with too few cases as inconclusive

What evidence proves who was allowed to act at the time?

A dated link between the person or automation that acted, the role or approval that permitted the action, and the policy version that required it. A closed ticket proves none of these; it proves that someone set a status.

NIST's separation of duties control, AC-5 in SP 800-53 Revision 5, asks organizations to identify and document duties that must be separated and to define access authorizations that support that separation, and its guidance notes that separation of duties violations can span systems. That is why the authority check cannot rely on the ticketing system alone: the approval may sit in the ticket, the role in the identity provider and the action in an endpoint console. The related non-repudiation control, AU-10, describes evidence that an individual performed an action such as approving information.

What evidence proves who was allowed to act at the time?
Evidence fieldTypical sourceWhat the check asksWhat a failure means
Actor id and actor typeEndpoint, firewall or SOAR audit logWas this a person, a playbook or a service account?Automation acting where a person was required
Actor role at action timeIdentity provider group change logDid the actor hold the required role on that date?Role granted later, or held only by assumption
Approval recordTicket approval field or change recordWas approval given before the action, by an eligible approver?Approval after the fact or by the actor themself
Policy version in forceVersioned policy or runbook pageWhich rule required the approval on that date?Case scored against the wrong version
Emergency override flagBreak-glass log or incident declarationWas an emergency path invoked and recorded?Unrecorded override; mark as exception
Action outcomeContainment, ticket closure, alert dispositionDid the action complete, and was it later reversed?Reversed actions need their own label

Worked example (illustrative): host isolation approval moves to the incident lead

This example is illustrative; the counts and durations are invented to show the arithmetic, not taken from any company. A security team exports nine months of incident cases, 1,200 in total. In month six its containment policy changes: isolating a production server now needs approval from the on-call incident lead, where before any tier-two analyst could isolate it alone.

The team sets one cut at the policy's effective time and measures that 90 percent of incidents in the archive closed within 10 days, so it uses a 10-day gap. Cases decided in months one to five, 640 of them, form the training side. Cases decided from the end of the gap through month eight, 410 of them, form the test side, and month nine, 110 cases, is held back. The remaining 40 cases were open across the cut and go to quarantine.

The authority check then runs on every isolation action in the test side. Of 85 isolations, 71 have an approval from someone in the incident-lead group on the date of the action. Nine were approved after the host was already isolated, and five have no approval at all, two of them under a recorded emergency declaration. The 71 become answer keys; the 14 others stay in the dataset as labeled exceptions so a model can be tested on recognizing a missing approval rather than learning to copy it.

The useful score is not overall accuracy. It is whether, on test cases involving production servers, the model now asks for incident-lead approval before isolation, compared with the same model's behavior on the training-era version. If the test side holds only a few dozen such cases, the team reports the difference with an interval and treats it as directional.

What must come out of security records before they leave the team?

Anything that would help an attacker or expose a credential, and anything that identifies people or customers beyond what the use requires. The decision evidence that makes a case useful, such as the alert, the triage reasoning, the approval and the outcome, can usually survive this if redaction replaces values rather than deleting fields.

  • Credentials, API keys, session tokens and secrets pasted into tickets or chat, which should also be rotated if found.
  • Exact detection logic and thresholds for rules still in production, which reveal what the team does not see; keep the rule id and revision instead.
  • Internal hostnames, IP ranges and network layouts, replaced with consistent pseudonyms so the same host stays the same token across a case.
  • Exploit steps, proof-of-concept code and unpatched vulnerability details that remain actionable.
  • Personal data about employees and customers, including names in phishing reports and screenshots of mailboxes.
  • Client telemetry a managed security provider holds under contract for someone else, which needs that client's permission, not only the provider's.

When does a time split mislead, and when should you stop?

A time split misleads when the timeline itself is unreliable. Stop and get a security reviewer and a statistician involved before relying on results in these cases.

  • Dispositions were backfilled in bulk after a tooling migration, so decision times reflect the cleanup date rather than the decision.
  • A policy change was applied retroactively, for example re-closing old incidents under new categories, so earlier cases now look like they followed the new rule.
  • The change affects too few cases after the cut to tell a real difference from noise; report it as inconclusive rather than widening the window until it looks significant.
  • Many cases were closed by automation, so the record shows a playbook's behavior rather than an analyst's reasoning; score those separately.
  • Role history is missing for the period, so authority can only be marked unverifiable; that may be fine for a training set but not for an answer key.
  • The records belong to clients of a managed service, so rights questions need counsel before any split is built.

Preparing versioned security operations history for licensing

Case archives that carry rule revisions, runbook versions and authority evidence are more useful than bare alert exports, because they let a buyer test whether a model adapts when the rules change. Before any of it is shared, separate the questions of who can access the data, whether privacy rules permit the use, who owns it and whether contracts allow licensing it; redaction answers none of those on its own.

Security teams that decide to go ahead can rely on SourceX for the rights review, contracting, data preparation and delivery that turn chosen operational records into a licensed AI dataset. The starting point is a short assessment with no data upload, and the company's approval of scope and terms, with its signature, comes before any delivery.

Frequently asked questions

Is a random train and test split ever acceptable for SOC case data?

Only for questions that do not depend on time or policy, such as checking a parser. For any test of how analysts or models make decisions, random splits leak incidents, rule versions and alert bursts across the boundary. Use a split based on decision time and policy versions instead.

How long should the gap after a cut be?

Base it on how long your incidents stay open, not on a fixed number. A common starting point is the duration within which most incidents in the archive closed. Cases open across the cut plus the gap go to quarantine rather than to either side.

What if our detection rules have no version history?

Then rule changes cannot be used as cut points, and the cases should carry an unknown rule revision rather than a guessed one. Runbook and policy changes with dated versions can still support a split. Start versioning rules now so future exports are usable.

Do we need to include the rule text itself?

Usually not. The rule id, revision and effective dates are enough to assign versions and to score answers against the policy in force. Exact logic for rules still in production is better left out because it shows what the team cannot see.

Should cases closed by automation be scored the same way?

No. A playbook closure reflects the playbook's logic, so score automated and human decisions separately and record the playbook version as the version in force. Mixing them hides whether the model learned analyst reasoning or automation behavior.

Sources

  • scikit-learn documents that ordinary cross-validation is inappropriate for time-ordered data and that its TimeSeriesSplit has a gap parameter that drops samples from the end of each training window. Source
  • The Sigma rule specification defines rule status values, creation and last-modified dates, and related-rule links such as derived, merged and obsolete. Source
  • OpenTelemetry semantic conventions define a security_rule.version attribute for the rule revision used in an analysis. Source
  • NIST SP 800-53 Revision 5 controls IR-4 (lessons learned into procedures), AC-5 (separation of duties) and AU-10 (non-repudiation), from NIST's official OSCAL catalog. Source
  • NIST SP 800-61 Revision 3, finalized April 3, 2025, supersedes Revision 2 and is organized as a CSF 2.0 Community Profile. Source
  • NIST explains that incident response procedures change too often and vary too much to maintain in a single static publication. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies