Privacy and preparation
Security operations dataset time splits across policy changes
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Split security operations records by the policy version in force, not at random. Cut the timeline at each change to a detection rule, runbook or approval policy, train on cases decided before it and test on cases decided after it, with a gap so open incidents do not straddle the cut. Then confirm every action was taken by someone authorized on that date.
Evaluation split
Security operations dataset
Key takeaways
- Random splits leak incidents, alert bursts and runbook versions across the boundary, so they cannot test whether reasoning adapts to a policy change.
- Cut at the effective time of changes that alter the expected action, and quarantine incidents open across the cut plus a gap based on your own case durations.
- Sigma's modified field keeps only the last change date, so rule versions in force must come from repository or SIEM history.
- Every action needs a dated authority check against role history, approvals and the policy version, because a closed ticket proves only a status change.
- Cases that fail the authority check stay in the dataset as labeled exceptions rather than answer keys.
- Redact secrets, live detection logic and internal network details by replacing values, so the decision evidence survives.
How do you evaluate security workflow reasoning across policy changes?
Evaluate it on incidents that were decided after a policy change the model never saw in training. Order every case by when it was decided, cut the timeline at the effective date of each detection rule, runbook or approval policy change, train on the earlier side and test on the later side, and score whether the model follows the version that was in force for each test case.
A random split cannot measure this. Alerts from one incident, near-duplicate alerts from one noisy rule, and cases handled under the same runbook revision land on both sides, so a model can score well by recalling the old procedure. scikit-learn's own documentation for its time-series splitter says ordinary cross-validation is inappropriate for time-ordered data because it trains on the future and evaluates on the past.
The second half of the test is authority. A security record shows what someone did, not whether they were allowed to do it. A case where an analyst isolated a production host without the approval the policy required is a poor answer key even if the outcome looked right, so each action needs an authority check against the roles and approvals that applied on that date.
Which changes count as a policy change in a security operations record?
Any change that would make a correct decision on one date wrong on another. In practice four kinds of change matter, and each one keeps its version in a different place, so the first job is to find where each version history lives before anyone chooses cut points.
Detection logic is the most visible. The Sigma rule specification gives rules a creation date and a modified date, a status such as experimental, test, stable or deprecated, and a related field that marks a rule as derived from, merged into or made obsolete by another. The modified field holds only the last modification date, so it cannot tell you which logic was live in a given month; the repository or SIEM change history has to. OpenTelemetry's semantic conventions also define a security_rule.version attribute for the rule revision used in an analysis, which is the kind of field to keep on every alert.
Procedures change because they are meant to. NIST SP 800-53 control IR-4 asks organizations to feed lessons learned from incident handling back into procedures, training and testing. NIST's incident response guidance, revised in April 2025 as SP 800-61 Revision 3, deliberately leaves out step-by-step procedures because those details change too often and vary too much between organizations, so a team's own versioned runbooks are the authority for what was expected at the time.
| Change type | Where the version usually lives | Field to keep on each case | Effect on the split |
|---|---|---|---|
| Detection rule logic or threshold | Rule repository history or SIEM rule audit log | Rule id, rule revision, revision effective time | Cut at each material revision of rules that drive many cases |
| Rule retired or replaced | Rule status and related-rule links | Superseding rule id, retirement time | Alerts from the retired rule stay on the training side |
| Runbook or playbook revision | Wiki page history, SOAR playbook versions | Runbook id and revision cited by the case | Test only on cases opened after the revision went live |
| Approval or escalation policy | Policy document versions, change tickets | Policy version, approver role required | Cut at the effective date; validate authority per version |
| Role or group membership | Identity provider audit log, access reviews | Actor role at action time, source of that fact | No cut, but feeds the authority check on every action |
| Tooling or data source change | Change management records, connector configs | Telemetry source and parser version | Record as a covariate; cut only if fields or meaning changed |
Where should the cut points go, and how wide should the gap be?
Put each cut at the effective time of a change that alters the expected action, and leave a gap after it at least as long as the time most incidents stay open. Changes that do not alter the expected action, such as a reworded runbook step, do not justify a cut and only shrink the test set.
The gap matters because security cases are not points in time. An incident opened under the old policy and closed under the new one can carry evidence from both sides, and a model trained right up to the cut may have seen the beginning of a case it is later tested on. scikit-learn's splitter has a gap parameter for the same reason: it drops samples from the end of each training window before the test window starts. Choose the gap from your own case durations, for example the time within which most incidents in the archive were closed, not from a default.
- List candidate changes with effective timestamps, not approval dates; a policy approved on one day and switched on a week later is cut on the second date.
- Keep a change as a cut only if a reviewer can state, in one sentence, an action that was correct before and wrong after.
- Prefer one strong cut with enough test cases over several weak cuts that leave each test window too thin to compare.
- Assign every case to a side by the time of the decision being scored, and quarantine cases whose open period crosses the cut plus the gap.
- Keep a later window after the newest change untouched as a final holdout, so repeated tuning on the first test window does not leak into the reported score.
The time-scoped split and authority validation protocol
The protocol below is the asset: a step list a security operations lead and a data lead can run together on an exported case archive. Each step names its input, the rule applied and what happens when the check fails. Run it on a sample before the full archive so problems with timestamps or missing version history surface early.
| Step | Input | Rule applied | Check or output | If it fails |
|---|---|---|---|---|
| 1. Freeze the case set | Case export with ids and timestamps | One row per case; alerts nested under it | Case count and date range recorded | Deduplicate merged incidents before going on |
| 2. Build the change log | Rule, runbook, policy and role histories | Every change has an effective timestamp and author | Versioned change table | Changes without a timestamp are excluded as cuts |
| 3. Choose cut points | Change table and case volumes per window | Expected action differs before and after | Cut list with one-line rationale each | Merge or drop cuts with too few later cases |
| 4. Assign sides | Cases, cuts, gap length | Decision time decides the side | Train, test, final holdout and quarantine labels | Cases missing a decision time go to quarantine |
| 5. Attach versions | Each case and change table | Rule revision, runbook revision and policy version in force | Version fields filled on every case | Unknown versions stay unknown, never inferred |
| 6. Validate authority | Actions, approvals, role history | Actor held the required role or approval at action time | Pass, fail or unverifiable per action | Failed cases are kept as labeled exceptions, not answer keys |
| 7. Check leakage | Train and test sides | No shared incident ids, hosts or near-duplicate alert bursts across the cut | Overlap report | Move overlapping clusters to quarantine |
| 8. Screen for secrets | Case text and attachments | No credentials, keys, internal IPs or exploit steps | Redaction log per case | Drop the case if redaction removes the decision evidence |
| 9. Score by version | Model outputs on test cases | Answer judged against the version in force for that case | Accuracy before and after each cut, with interval | Report cuts with too few cases as inconclusive |
What evidence proves who was allowed to act at the time?
A dated link between the person or automation that acted, the role or approval that permitted the action, and the policy version that required it. A closed ticket proves none of these; it proves that someone set a status.
NIST's separation of duties control, AC-5 in SP 800-53 Revision 5, asks organizations to identify and document duties that must be separated and to define access authorizations that support that separation, and its guidance notes that separation of duties violations can span systems. That is why the authority check cannot rely on the ticketing system alone: the approval may sit in the ticket, the role in the identity provider and the action in an endpoint console. The related non-repudiation control, AU-10, describes evidence that an individual performed an action such as approving information.
| Evidence field | Typical source | What the check asks | What a failure means |
|---|---|---|---|
| Actor id and actor type | Endpoint, firewall or SOAR audit log | Was this a person, a playbook or a service account? | Automation acting where a person was required |
| Actor role at action time | Identity provider group change log | Did the actor hold the required role on that date? | Role granted later, or held only by assumption |
| Approval record | Ticket approval field or change record | Was approval given before the action, by an eligible approver? | Approval after the fact or by the actor themself |
| Policy version in force | Versioned policy or runbook page | Which rule required the approval on that date? | Case scored against the wrong version |
| Emergency override flag | Break-glass log or incident declaration | Was an emergency path invoked and recorded? | Unrecorded override; mark as exception |
| Action outcome | Containment, ticket closure, alert disposition | Did the action complete, and was it later reversed? | Reversed actions need their own label |
Worked example (illustrative): host isolation approval moves to the incident lead
This example is illustrative; the counts and durations are invented to show the arithmetic, not taken from any company. A security team exports nine months of incident cases, 1,200 in total. In month six its containment policy changes: isolating a production server now needs approval from the on-call incident lead, where before any tier-two analyst could isolate it alone.
The team sets one cut at the policy's effective time and measures that 90 percent of incidents in the archive closed within 10 days, so it uses a 10-day gap. Cases decided in months one to five, 640 of them, form the training side. Cases decided from the end of the gap through month eight, 410 of them, form the test side, and month nine, 110 cases, is held back. The remaining 40 cases were open across the cut and go to quarantine.
The authority check then runs on every isolation action in the test side. Of 85 isolations, 71 have an approval from someone in the incident-lead group on the date of the action. Nine were approved after the host was already isolated, and five have no approval at all, two of them under a recorded emergency declaration. The 71 become answer keys; the 14 others stay in the dataset as labeled exceptions so a model can be tested on recognizing a missing approval rather than learning to copy it.
The useful score is not overall accuracy. It is whether, on test cases involving production servers, the model now asks for incident-lead approval before isolation, compared with the same model's behavior on the training-era version. If the test side holds only a few dozen such cases, the team reports the difference with an interval and treats it as directional.
What must come out of security records before they leave the team?
Anything that would help an attacker or expose a credential, and anything that identifies people or customers beyond what the use requires. The decision evidence that makes a case useful, such as the alert, the triage reasoning, the approval and the outcome, can usually survive this if redaction replaces values rather than deleting fields.
- Credentials, API keys, session tokens and secrets pasted into tickets or chat, which should also be rotated if found.
- Exact detection logic and thresholds for rules still in production, which reveal what the team does not see; keep the rule id and revision instead.
- Internal hostnames, IP ranges and network layouts, replaced with consistent pseudonyms so the same host stays the same token across a case.
- Exploit steps, proof-of-concept code and unpatched vulnerability details that remain actionable.
- Personal data about employees and customers, including names in phishing reports and screenshots of mailboxes.
- Client telemetry a managed security provider holds under contract for someone else, which needs that client's permission, not only the provider's.
When does a time split mislead, and when should you stop?
A time split misleads when the timeline itself is unreliable. Stop and get a security reviewer and a statistician involved before relying on results in these cases.
- Dispositions were backfilled in bulk after a tooling migration, so decision times reflect the cleanup date rather than the decision.
- A policy change was applied retroactively, for example re-closing old incidents under new categories, so earlier cases now look like they followed the new rule.
- The change affects too few cases after the cut to tell a real difference from noise; report it as inconclusive rather than widening the window until it looks significant.
- Many cases were closed by automation, so the record shows a playbook's behavior rather than an analyst's reasoning; score those separately.
- Role history is missing for the period, so authority can only be marked unverifiable; that may be fine for a training set but not for an answer key.
- The records belong to clients of a managed service, so rights questions need counsel before any split is built.
Preparing versioned security operations history for licensing
Case archives that carry rule revisions, runbook versions and authority evidence are more useful than bare alert exports, because they let a buyer test whether a model adapts when the rules change. Before any of it is shared, separate the questions of who can access the data, whether privacy rules permit the use, who owns it and whether contracts allow licensing it; redaction answers none of those on its own.
Security teams that decide to go ahead can rely on SourceX for the rights review, contracting, data preparation and delivery that turn chosen operational records into a licensed AI dataset. The starting point is a short assessment with no data upload, and the company's approval of scope and terms, with its signature, comes before any delivery.
Frequently asked questions
Is a random train and test split ever acceptable for SOC case data?
Only for questions that do not depend on time or policy, such as checking a parser. For any test of how analysts or models make decisions, random splits leak incidents, rule versions and alert bursts across the boundary. Use a split based on decision time and policy versions instead.
How long should the gap after a cut be?
Base it on how long your incidents stay open, not on a fixed number. A common starting point is the duration within which most incidents in the archive closed. Cases open across the cut plus the gap go to quarantine rather than to either side.
What if our detection rules have no version history?
Then rule changes cannot be used as cut points, and the cases should carry an unknown rule revision rather than a guessed one. Runbook and policy changes with dated versions can still support a split. Start versioning rules now so future exports are usable.
Do we need to include the rule text itself?
Usually not. The rule id, revision and effective dates are enough to assign versions and to score answers against the policy in force. Exact logic for rules still in production is better left out because it shows what the team cannot see.
Should cases closed by automation be scored the same way?
No. A playbook closure reflects the playbook's logic, so score automated and human decisions separately and record the playbook version as the version in force. Mixing them hides whether the model learned analyst reasoning or automation behavior.
Sources
- scikit-learn documents that ordinary cross-validation is inappropriate for time-ordered data and that its TimeSeriesSplit has a gap parameter that drops samples from the end of each training window. Source
- The Sigma rule specification defines rule status values, creation and last-modified dates, and related-rule links such as derived, merged and obsolete. Source
- OpenTelemetry semantic conventions define a security_rule.version attribute for the rule revision used in an analysis. Source
- NIST SP 800-53 Revision 5 controls IR-4 (lessons learned into procedures), AC-5 (separation of duties) and AU-10 (non-repudiation), from NIST's official OSCAL catalog. Source
- NIST SP 800-61 Revision 3, finalized April 3, 2025, supersedes Revision 2 and is organized as a CSF 2.0 Community Profile. Source
- NIST explains that incident response procedures change too often and vary too much to maintain in a single static publication. Source
See if your company qualifies
A short company assessment. No data uploads are needed.