AI uses for records
Incident dataset recovery experiment: held-out design and stop rules
By SourceX Editorial · Updated
10 sections
Short answer
An incident dataset recovery experiment asks whether giving an agent a company's past incident records makes its recovery faster and safer on incidents it has never seen. Replay held-out incidents in a fixed sandbox with bounded actions, compare records with no records, stripped records and a length-matched placebo, measure time to verified recovery and harmful actions, and stop on preset rules. No results are published; this is the design.
Record timeline
Incident dataset recovery
Key takeaways
- Only a replay can answer the question: the same held-out incident, run once per arm, in a sandbox that checks steady state itself.
- Four arms separate what records say from how much the agent reads: none, full, outcomes removed, and a length-matched placebo.
- Count failed runs at the budget cap and track harmful actions, or a faster arm can win by breaking things.
- In an illustrative simulation, 48 incidents with 3 runs each reached 85.8 percent power, against 47.4 percent for 12 incidents with 10 runs.
- A public sandbox such as the OpenTelemetry demo offers 19 failure scenarios; real held-out power needs a company's own incidents.
What question does an incident dataset recovery experiment answer?
Whether giving an agent a company's past incident records makes its recovery decisions faster and safer on incidents it has never seen. Written as a testable claim: with the agent, tools and budget held fixed, access to records from before a cutoff date cuts the time to verified recovery on later incidents by at least a margin set in advance, without increasing actions that make the outage worse. The answer tells a buyer whether incident records are worth licensing, which fields carry the value and how many incidents an evaluation needs.
Production history cannot settle it. Teams that keep good records tend to differ from teams that do not, every incident is different, responders learn as they go, and recovery time depends on load and luck. There is no second version of the same outage handled without the records. The only clean comparison replays the same incident several times in a fixed environment, once per arm, and that is what the design below does.
No such experiment has been run for this article, by SourceX or with SourceX data. What follows is the design, two computations that size it, and the rules that would decide what a result means.
The preregistration sheet: ten decisions fixed before the first run
Ten choices decide whether a result can be trusted, so they are written down before any run and not changed afterwards. The margin, budget and action list below are illustrative defaults; a buyer sets its own, but sets them first.
At a glance
The preregistration sheet: ten decisions fixed before the first run
Each row gives the choice and the reason it is made that way:
| Decision | Specification | Why it is set this way |
|---|---|---|
| Hypothesis | Records cut median time to verified recovery by at least 20 percent on held-out incidents, with no rise in harmful actions | A margin fixed in advance keeps a small, noisy gain from being presented as value |
| Unit | One held-out incident, replayed in every arm | Pairing removes incident difficulty from the comparison |
| Held-out rule | Incidents after a cutoff date, minus any whose own postmortem or a near-copy sits in the records | Stops the agent from recognizing the answer instead of reasoning to it |
| Environment | A sandbox that injects each incident's fault and checks steady state automatically | Every arm and run starts from the same verified state |
| Action set | A fixed allowlist: restart, scale, roll back a deploy, change a config value, wait, escalate | Bounded actions make invalid and harmful moves countable |
| Arms | No records; full records; records with outcomes removed; a length-matched placebo | Separates what the records say from how much text the agent reads |
| Primary metric | Time to verified recovery, capped at a budget; unrecovered runs score the cap | Failed runs stay in the analysis instead of dropping out |
| Secondary metrics | Invalid actions, harmful actions, and incidents recovered in every run | Speed bought with risk is not an improvement |
| Analysis | Paired difference in mean log time per incident; two-sided test at 0.05; one interim look | The test and its single early look are fixed before data |
| Stop rules | Harm, validity, leakage and futility, defined in their own section | Decided before anyone has a reason to bend them |
Which incidents can be held out, and why one public sandbox is too small
Incidents qualify for the held-out set by date and by mechanism, and a single public sandbox offers too few mechanisms to test much. Split by time first: every incident after the cutoff is a candidate, every record before it may go to the agent. Then remove leaks, meaning any held-out incident whose own postmortem, runbook edit or a near-copy of its timeline is in the records pool. Coding benchmarks use the same guard; LiveCodeBench, for example, scores models only on problems published after their training cutoff to avoid contamination.
Keep a second label on each held-out incident: whether the records hold a precedent with the same failure mechanism. If records help through precedent, the gain should be larger where one exists. A gain just as large on novel incidents points to something generic, such as a better habit of checking recent deploys, which a cheaper document might also supply.
Public sandboxes show the limit. The OpenTelemetry Demo, a multi-service application with cart, checkout and payment services written in 12 languages, ships its fault scenarios as feature flags. At commit 8f640de of 27 October 2026, its flag file defines 20 flags; one is labeled not a failure scenario, leaving 19 failures with 38 intensity settings between them, and 7 of the 19 offer graded intensities. The project's documentation names 11 services those flags affect, and its flag table leaves out 3 of the 20 configured flags, 2 of them failure scenarios, while listing 2 flags the configuration lacks, so a scenario list should be read from the configuration at a stated commit.
Nineteen scenarios make a good pilot and a thin base for a held-out test. Hold out a third and about six remain, and the planning simulation below shows how little six incidents can detect. The fault flags themselves also have to sit outside the agent's action set; otherwise the fix becomes a lookup. Real held-out power comes from turning a company's own past incidents into replayable scenarios.
Four arms, and what each comparison isolates
Four arms answer three different buyer questions, and the comparisons matter more than any single arm's score. Every arm uses the same agent, tools, budget and incidents; only the material the agent can read changes.
At a glance
Four arms, and what each comparison isolates
The arms, and the comparison each one makes possible:
- No records: the agent sees alerts, dashboards and the action set. This is the baseline every other arm is measured against.
- Full records: the same, plus pre-cutoff postmortems, incident timelines, runbook versions and recovery notes. Full against none gives the total effect of having the archive.
- Records with outcomes removed: the same documents with what was tried, what worked and how recovery was verified stripped out. Full against stripped measures the value of outcome fields, which decides what a buyer should insist on.
- Length-matched placebo: unrelated engineering text of the same length. Full against placebo shows whether the gain comes from the records' content or simply from reading more.
Metrics: time to verified recovery, and the moves that make things worse
The primary measure is time to verified recovery, and verified means the environment's own steady-state check passes, not that the agent says it is done. The Principles of Chaos Engineering define steady state as a measurable output of a system that indicates normal behavior, such as throughput, error rates or latency percentiles; a recovery check should test the same outputs the fault disturbed. Time runs in simulated minutes or environment steps from the first alert.
Runs that never recover stay in the data at the budget cap, because dropping them would reward an arm that fails quietly. A few very long runs can dominate a plain mean, so the analysis compares log times per incident and reports medians and the share recovered within budget instead.
Speed is not enough on its own. An agent that restarts every service may recover quickly and break three other things, so the secondary measures count the moves that should not happen:
- Invalid actions: attempts outside the allowlist, which the environment rejects and logs.
- Harmful actions: allowed actions after which a steady-state metric gets worse, counted per run.
- Consistency: the share of held-out incidents recovered in every run, since an agent that recovers an incident one time in three is not reliable on it.
- Agreement with the record: whether the agent's recovery path matches the one the human team used, kept as an audit measure, not a score, because a different path can be just as good.
How many incidents and runs? An illustrative planning simulation
Under the assumptions below, adding held-out incidents buys more certainty than adding runs of the same incidents, and six incidents detect little. We simulated the paired design with a fixed seed: each incident has its own difficulty, records help some incidents more than others, and each run adds noise. Every input is an assumption chosen for illustration, not a measured property of any agent or archive.
The model: log recovery time varies across incidents with standard deviation 0.8, the records effect varies across incidents with standard deviation 0.3, and run-to-run noise has standard deviation 0.5. The true effect is a 20 percent cut in recovery time. Each cell ran 4,000 simulated experiments, and power is the share that found a significant cut at 0.05. With no true effect, the test wrongly favored records in 2.3 to 3.0 percent of experiments across cells, close to the 2.5 percent expected for one direction of a two-sided test.
At similar effort, more incidents win: 48 incidents with 3 runs per arm, 144 runs, reach 85.8 percent power, while 12 incidents with 10 runs, 120 runs, reach 47.4 percent. The reason is the uneven effect across incidents, which more runs of the same incident cannot average away. Power by held-out incidents and runs per arm (Illustrative):
| Held-out incidents | 1 run per arm | 3 runs per arm | 10 runs per arm |
|---|---|---|---|
| 6 | 8.8% | 13.8% | 22.7% |
| 12 | 14.9% | 28.8% | 47.4% |
| 24 | 26.9% | 53.5% | 79.7% |
| 48 | 49.3% | 85.8% | 98.2% |
Stop rules written down before anyone sees a result
Four stop rules end or pause the experiment, and all four are fixed before the first run. Fault-injection tools build the same habit in: AWS Fault Injection Service ties an experiment to stop conditions defined from the application's steady state, stops the experiment when one is triggered, and does not let a stopped experiment resume.
At a glance
Stop rules written down before anyone sees a result
The interim look needs the most care. FDA's 2019 guidance on adaptive clinical trial designs shows that testing at an interim look and again at the end, each at the usual significance level, pushes the overall false-positive rate above the nominal level, and that methods exist to set interim and final thresholds together so the overall rate stays controlled. The same arithmetic applies to an agent experiment that peeks halfway.
The rules:
- Harm stop: any action outside the allowlist executes, or any change reaches outside the sandbox. Halt, fix the guardrail, and restart every arm from zero.
- Validity stop: an audit finds the recovery check passing on a broken state, or the no-records arm recovers most incidents almost at once. The environment is measuring the wrong thing or is too easy; rebuild it before going on.
- Leakage stop: the records arm's gains cluster on incidents with near-copies in the pool. Re-run the leak screen and rebuild the held-out set.
- Futility look: one interim analysis at half the incidents. If the upper confidence bound on the improvement is below the preset margin, stop and report a null. No early stop for success; the full set runs, and the result is published whichever way it falls.
The strongest objection: a passing check is not a recovered business
The experiment measures recovery in a replay, not in a business, and three gaps follow from that. First, a sandbox check can pass while customers are still affected, because the check tests the outputs someone thought to write down. Second, only incidents that can be replayed get in, and the hardest ones, such as vendor outages, data corruption or slow partial failures, are the least replayable. Third, records may help a human responder in ways a sandbox cannot show, such as knowing whom to call.
None of these sinks the design, but each limits what a result can claim. Report which incident types were excluded as not replayable, audit a sample of passing runs against what the original postmortem says customers saw, and state plainly that a gain in the replay is a gain in the replay. A buyer deciding whether to license records should treat a positive result as necessary evidence, not sufficient.
What a buyer should request, and what has not been published
To run this design, a buyer needs incident records that can become replayable scenarios and an archive deep enough to split by date. That means, for each incident, the alerts and their timestamps, a timeline of actions with who took them, the configuration or deploy changes involved, how recovery was verified, the postmortem and the runbook version in force, plus enough of the service map to rebuild the failure without live credentials or customer data.
No result from this design has been published, and SourceX has not run it. Datasets sourced through SourceX are found on request rather than held in stock, so a buyer can describe the scenarios, fields and date range it needs and attach this experiment as the acceptance test a delivery would have to support. Whether incident records improve recovery decisions is exactly the kind of claim that should be settled by a preregistered test, not asserted by a seller.
Methods note: sandbox inventory, simulation and sources
Sandbox inventory: src/flagd/demo.flagd.json in github.com/open-telemetry/opentelemetry-demo at commit 8f640de (27 October 2026), read by our script. Every key under flags counts as a flag; a flag is a failure scenario unless its description contains the text not a failure scenario (one flag, dropped and printed); intensity settings are variants other than off. Services come from the Service(s) column of the feature flag table in content/en/docs/demo/feature-flags/_index.md of github.com/open-telemetry/opentelemetry.io at commit 67559fc; 3 of the 20 configuration flags (2 failure scenarios and the non-failure flag) are missing from that table, and two table rows are missing from the configuration.
Simulation: normal draws from Python's random.gauss on random.Random seeded with 20261027 + 1000 x incidents + runs for each cell (that seed + 7 for the no-effect check), 4,000 experiments per cell, draws ordered incident by incident as difficulty, effect variation, then arm-0 runs and arm-1 runs. Each experiment applies a two-sided one-sample t-test to the per-incident differences in mean log time and counts a success when p is below 0.05 and the difference favors records. All variance inputs and the 20 percent effect are illustrative. The figures reproduce only with random.gauss: with random.normalvariate and the same seeds, 48 incidents with 3 runs give 84.6 percent instead of 85.8. Shares come from integer counts, rounded half up once on the exact fraction.
Outside sources: the Principles of Chaos Engineering, AWS Fault Injection Service documentation, FDA's 2019 adaptive design guidance and the LiveCodeBench paper, each verified through a search excerpt on its own domain. Not measured: any agent's recovery performance, any company's incident archive, and the effect of records on human responders. Scripts and outputs are kept in the writer's scratch folder: bin/otel_scenarios.py and bin/recovery_power_sim.py, with the corrected copies that produced the published figures, bin/otel_scenarios_v2.py (all 20 flags checked against the docs table) and bin/recovery_power_sim_v2.py (half-up rounding, generator check).
Frequently asked questions
How do you test whether incident records help an AI agent recover from outages?
Replay incidents that happened after a cutoff date in a fixed sandbox, once with the records and once without, plus arms with outcomes stripped and with unrelated text of equal length. Compare time to verified recovery and harmful actions for each incident across arms.
What is a held-out incident?
An incident the agent's records cannot contain: it happened after the records' cutoff, and neither its postmortem nor a near-copy of its timeline is in the pool. Holding incidents out this way keeps the agent from recognizing an answer instead of working one out.
Why not measure mean time to recovery?
A few very long incidents can dominate a mean, and runs that never recover would drop out unless they are scored at a cap. Comparing log times per incident and reporting medians and the share recovered within budget keeps every run and resists outliers.
How many incidents does a recovery experiment need?
It depends on how much the effect varies across incidents. In an illustrative simulation with a 20 percent true effect, 48 held-out incidents with 3 runs each gave 85.8 percent power, while 6 incidents gave under 23 percent even with 10 runs.
Has anyone published results from this design?
Not in this article. SourceX has not run the experiment and reports no result; the page publishes the design so a buyer can use it as an acceptance test for incident data.
Sources
- The OpenTelemetry demo's flag file at commit 8f640de defines 20 flags, 19 of them failure scenarios with 38 intensity settings. Source
- OpenTelemetry's documentation says the demo's feature flags simulate scenarios and lists the services each flag affects. Source
- The Principles of Chaos Engineering define steady state as a measurable system output that indicates normal behavior, such as throughput, error rates or latency percentiles. Source
- AWS Fault Injection Service stops an experiment when a stop condition defined from steady state is triggered, and a stopped experiment cannot be resumed. Source
- FDA's 2019 adaptive design guidance shows that testing at an interim and a final look at the conventional level inflates the overall Type I error rate. Source
- LiveCodeBench evaluates models on problems released after their training cutoff to detect and avoid contamination. Source
- OpenTelemetry's demo documentation lists services written in 12 languages, including cart, checkout and payment services. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.