MSP RMM alert-to-remediation records for IT automation agents

What an RMM remediation trajectory contains

A usable trajectory joins three systems that MSPs rarely export together: the RMM (alerts, monitors, script runs), the PSA (tickets, time entries, notes) and the patch or backup console. Typical RMM platforms include ConnectWise Automate, NinjaOne, Datto RMM, N-able N-central and Kaseya VSA; PSA tools include ConnectWise PSA, Autotask and HaloPSA. Each stores alert IDs, device IDs and ticket IDs differently, so the join key is the first thing to check in a sample.

The minimum fields per trajectory are:

• Trigger: monitor type and threshold (for example C: free space below 10%, Windows service Spooler stopped, Windows Update error code, backup job status Failed), alert timestamp and severity.
• Context: OS and build, device role (workstation, domain controller, file server, hypervisor), agent version, client tier as a token, and prior alerts on the same device in the last 30 days.
• Actions: ordered script runs and manual steps, with script name, language (PowerShell, Bash, batch), parameters, exit code, stdout/stderr excerpts and who or what launched it (automation policy or technician).
• Outcome: final monitor state, ticket status and close code, time to resolve, and whether the same alert fired again on that device.

This is the same thought-action-result structure researchers use to study software-engineering agents, where each step pairs an action with its observed result and success is defined by an outcome check [4]. For broader IT service data that stops at the ticket, see ITSM ticket datasets; this page covers the action-and-outcome layer underneath.

Why outcome labels need a recurrence window

Recurrence is a stronger quality signal than ticket closure, because MSP tickets are often closed by policy or when a monitor auto-clears. A disk-space alert "resolved" by a temp-file cleanup script that fires again three days later is a failed remediation, even if the ticket says Resolved. Label each trajectory as one of: auto-resolved by policy, resolved by technician, escalated, or recurred within N days (state N explicitly).

Ask suppliers how close codes are set and whether auto-close rules exist, because a close code chosen from a dropdown at 6 p.m. is weak ground truth. Our guide to verifying outcome labels in operational records covers the general tests; for RMM data, cross-check the ticket outcome against the monitor's state history rather than trusting either alone.

Secrets, tenant IDs and the RMM attack surface

RMM scripts must be secret-scanned and de-identified before delivery, because they routinely embed admin credentials, API tokens, Microsoft 365 tenant IDs, internal hostnames, UNC paths and public IPs. Evaluation tools already check fine-tuning samples for secrets and sensitive values before training [2], and models are known to memorize and reproduce rare strings such as keys present in training text [3]. A credential that reaches model weights can resurface in generated scripts.

The stakes are higher than ordinary PII. CISA, NSA and MS-ISAC have warned that attackers use legitimate RMM software for persistence and command and control, and that compromising one MSP can reach many of its customers [1]. Treat any delivered script that still points at a live endpoint, token or tenant as a security incident, not a data-quality defect.

Practical controls to require:

• Run pattern and entropy scanners (for example gitleaks or TruffleHog rule sets) over script bodies, parameters and stdout, then manually review a sample of hits and misses.
• Replace hostnames, domains, tenant GUIDs and IPs with consistent tokens so a trajectory still shows that the same server failed twice.
• Strip or neutralize download URLs and remote-execution endpoints so no script runs as delivered.
• Remove user names and emails from ticket notes; technician names become role tokens.

Multi-tenant rights: one MSP, many data owners

An MSP holds its clients' device and ticket data under each client's master services agreement, so the MSP alone may not be able to license all of it. Expect licensable subsets to be aggregated, limited to clients whose agreements permit it, or opt-in by client. Ask the supplier which clients are in scope, what their agreements say about data use, and how out-of-scope clients were filtered (by client ID at extraction, not by eyeballing).

Ticket notes are where scope leaks happen: a note on one client's ticket may quote another client's environment or an end user's email. Also check for regulated content, since healthcare clients' tickets can contain screenshots or notes with patient details that need their own review. If you want help scoping a multi-client request before approaching suppliers, see how SourceX works with AI data buyers or compare options to license IT service tickets.

Request template for RMM remediation data

A precise request lets suppliers give a clearer yes-or-no than a category name does. Use this as a starting brief.

Illustrative example: invented to show structure; it does not describe an available dataset.

One illustrative trajectory record:

{
  "trajectory_id": "t_000184",
  "alert": {"family": "patch_failure", "monitor": "windows_update", "error_code": "0x800f0922", "severity": "warning"},
  "context": {"os": "Windows Server 2019", "role": "file_server", "client_tier": "tier_B", "prior_alerts_30d": 2},
  "actions": [
    {"seq": 1, "actor": "policy", "script": "clear_softwaredistribution.ps1", "exit_code": 0},
    {"seq": 2, "actor": "policy", "script": "retry_cumulative_update.ps1", "exit_code": 1, "stderr_excerpt": "insufficient space on System Reserved"},
    {"seq": 3, "actor": "technician", "step": "extend recovery partition, rerun update", "exit_code": 0}
  ],
  "outcome": {"final_state": "healthy", "label": "technician_resolved", "recurred_within_14d": false, "minutes_to_resolve": 96},
  "ticket_note_redacted": "Update failed on reserved partition; resized and reapplied. Host [HOST_17]."
}

Using the records for SFT and agent evaluation

Technician-resolved trajectories are the most valuable SFT examples because they show the step automation could not take. Policy-resolved runs teach routine fixes but are easy to over-represent; balance by alert family and outcome. Failed attempts followed by a successful step are useful preference or contrastive data, since they show what not to run first.

For evaluation, hold out whole clients or time periods, not random trajectories, so the agent is not tested on near-duplicates of training runs. Real-environment benchmarks such as OSWorld score agents by executing tasks and checking resulting state [6], and tau-bench checks tool use against policies and database state [5]. RMM records let you build the same kind of check for IT work: replay the alert in a sandbox VM, let the agent act, and compare the end state and recurrence against the recorded outcome. Related patterns appear in exception-handling records, SOC alert triage decisions and telecom trouble tickets.

Common failure modes in RMM data

Most problems surface in the first sample review, if you look for them:

• Orphaned script runs: the RMM logs a script with no linked alert or ticket, so actions cannot be attributed to a trigger.
• Truncated output: stdout capped at a few kilobytes hides the real error.
• Silent auto-close: monitors clear on reboot, and the ticket closes as resolved with no fix.
• Template notes: technicians paste the same close note, so free text adds no signal.
• Script drift: the same script name changed content over time; ask for a version hash per run.

More on sample checks is in the training data quality guide, and the industries hub lists neighboring operational datasets. Buyers in this sector can also start from IT managed services buyers, IT operations agent data and IT incident recovery records.

Requesting RMM alert-to-remediation records from SourceX

SourceX sources operational datasets, including engineering and support records, from US companies on request; nothing is held in stock and a request does not guarantee a match. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, with personal details removed or replaced and every release approved by the supplying company. You describe the data you need, not the businesses, at SourceX for AI data buyers.