AI uses for records
Incident fragmentation dataset sampling: draw incidents, not tickets
By SourceX Editorial · Updated
9 sections
Short answer
Incident fragmentation dataset sampling starts by deciding what one task is. A single outage can leave dozens of alerts, duplicate customer tickets, chat threads and copied resolution notes, and counting each as its own example overstates sample size, narrows intervals and leaks answers across splits. Roll fragments up to an incident key, sample incidents first, cap fragments per incident, split by incident and report incident counts beside record counts.
Record timeline
Incident fragmentation
Key takeaways
- The sampling unit follows the decision being trained or tested: an incident for end-to-end resolution, a fragment only when it carries its own decision and success check.
- Alerting and help desk tools already group, link and copy records in configurable ways, so raw incident and ticket counts depend on settings that may have changed during the export window.
- With correlated fragments, the honest effective sample size sits between the number of incidents and the number of records; report both and cluster intervals by incident.
- Every fragment of one incident belongs on the same side of a training, validation or test split, or copied resolution text hands the model the answer.
- A coverage statement should name the incident count, the fragment distribution and the share of records held by the largest incidents, not just total rows.
Why does one long incident distort a sample?
Because samples usually count records, and a long incident produces many records that share one cause, one set of responders and often one resolution. In an illustrative export where a single outage left 140 alerts, tickets and chat threads, a random draw of records keeps landing on that outage, and each landing adds far less new information than its row suggests.
Incident fragmentation is the gap between the number of records an incident leaves behind and the number of independent decisions it contains. It causes three problems a dataset curator can check: sample size is overstated, so intervals come out too narrow; fragments of one incident land on both sides of a split, so a test rewards recognition instead of reasoning; and coverage looks broad when it is one bad week seen from many angles.
The fix is not deletion. Fragments carry real signal, such as how different customers described the same symptom or how an alert storm unfolded minute by minute. The fix is to decide what one task is before sampling, attach every record to an incident key, and let the incident, not the row, set the count that every claim rests on.
Where do incident fragments come from in alerting and ticketing exports?
They come from tools doing their jobs: alerting systems group some signals and not others, help desks link duplicate tickets to a parent, and responders talk in channels the ticket never sees. Each mechanism leaves a different trace, so the roll-up has to read each one rather than trust a single incident ID.
At a glance
Where do incident fragments come from in alerting and ticketing exports
PagerDuty's documentation for Intelligent Alert Grouping (checked 2026-10-27) describes a new alert joining an existing incident only when the previous grouped alert falls inside the time window, the incident is less than 24 hours old and the model judges the alerts similar; otherwise a new incident opens. Manual merges in the web app also train the model. A long outage can therefore surface as several incidents, and the incident count reflects grouping settings and responder habits as much as faults.
Zendesk's guide to problem and incident tickets (checked 2026-10-27) shows the opposite pattern. Duplicate customer tickets are linked as incidents to one problem ticket, and solving the problem sets every linked incident to solved and adds the problem's closing comment to those not already solved. The export then holds many tickets ending in the same resolution text, which a model will happily read as dozens of agreeing labels.
Five kinds of fragment show up in most incident archives:
- Alert fragments: repeated or re-triggered alerts for one cause, split across incidents when grouping windows or age limits expire.
- Customer ticket fragments: duplicate reports of one symptom, often linked to a parent or problem record and closed by its resolution.
- Conversation fragments: chat threads, bridge-call notes and status-page updates that narrate the same incident in pieces.
- Change and follow-up fragments: the rollback, the hotfix ticket, the postmortem and its action items, each with its own ID.
- Recurrence fragments: the same fault returning days later, logged as a new incident or as the old one reopened, depending on local practice.
Set the task boundary from the decision you are training
Choose the unit by asking which decision the model must make and how success will be checked. Anthropic's guide to agent evaluations (checked 2026-10-27) describes a task as one test whose inputs and success criteria are set out explicitly, and the outcome as the environment's final state once a trial ends. A fragment with no input of its own and no success check of its own is part of a task, not a task.
One incident archive can support several boundaries. A triage model may learn from individual alerts, although the alerts of one incident still share a cause; an agent that resolves incidents end to end needs the whole incident as one task. The table maps five common uses to a unit, the fragments that belong inside it and the key that keeps those fragments together.
| Intended use | Sampling unit | Fragments inside the unit | Group key for splits and intervals |
|---|---|---|---|
| Alert triage or routing | One alert, with a cap per incident | The alert payload and what responders knew at that moment | Incident key |
| End-to-end resolution by an agent | One incident | Alerts, tickets, chat, changes and the recovery evidence | Incident key, then time period |
| Customer replies during an outage | One customer ticket that needed its own reply decision | That ticket's thread, with the shared problem resolution as context | Problem or parent record |
| Duplicate and related-ticket linking | One pair of tickets | Both tickets and the evidence that links or separates them | Incident key of either ticket |
| Root-cause reasoning | One incident with a written cause | Timeline, change records, postmortem and the stated cause | Incident key plus recurrence chain |
The incident roll-up and sampling worksheet
Fill one worksheet per dataset before drawing anything, and attach it to the dataset card. Each line forces a decision a reviewer can check later, and the entries show the level of detail that makes a line useful.
The entries describe an invented 18-month IT operations export of 12,000 records that rolls up to 1,450 incidents; none are measured figures.
| Worksheet line | Question it settles | Evidence to attach | Illustrative entry |
|---|---|---|---|
| 1. Decision and unit | What one task is and how success is checked | Task definition with input, success check and outcome record | End-to-end resolution; one incident is one task |
| 2. Linking fields | Which fields tie fragments together | Parent, problem, grouping, merge and change references per system | Alerting incident ID, help desk problem link, change ticket reference |
| 3. Heuristic links | How unlinked fragments are attached | Rule by service, symptom and time window, plus a reviewed sample of its links | Same service and symptom within 6 hours; 200 links checked by two reviewers |
| 4. Fragment distribution | How uneven incidents are | Median, 90th percentile and maximum fragments per incident; share of records in the top 1 percent | Median 4, 90th percentile 18, maximum 412; the 15 largest incidents hold 18 percent of records |
| 5. Grouping history | Whether incident counts compare across periods | Dates of alert grouping and ticket-linking changes | Grouping window widened in month 9; periods reported separately |
| 6. Draw and cap | How many incidents, and how many fragments from each | Stratified incident draw, per-incident fragment cap, inclusion weights | Drawn by service and severity; at most 5 fragments each for fragment-level tasks |
| 7. Split rule | How fragments stay together | Split table keyed by incident and time | Whole incidents assigned; test set from the last 3 months only |
| 8. Reporting unit | Which count every claim uses | Results with incident counts, record counts and incident-clustered intervals | Reported as 1,450 incidents (12,000 records) |
How many independent cases does a fragmented export hold?
Somewhere between the number of incidents and the number of records, and where it falls depends on how alike the fragments of one incident are. If every fragment were a copy, the illustrative export would hold 1,450 cases; if fragments were unrelated, 12,000. Outcome labels copied from a parent resolution behave almost like copies, so the incident count is the safer planning number.
Treating the larger number as the sample is what clinical trial methodologists call a unit-of-analysis error. The Cochrane Handbook's chapter on cluster-randomized and other variant trials (checked 2026-10-27) warns that analyzing clustered observations as if they were independent yields confidence intervals that are too narrow and P values that are too small, which can produce false claims of an effect. Anthropic's note on the statistics of model evaluations (checked 2026-10-27) gives the evaluation version: when items are not independent, cluster the standard errors on the unit that was sampled.
In practice, plan the sample with incidents as the unit, compute intervals clustered by incident and print the incident count next to every record count. A headline of 12,000 examples, when 15 incidents supply 18 percent of them, should not survive review.
When the task really is fragment-level, such as alert routing, report two views of the same results. The traffic view weights every fragment equally and shows how the model performs on the volume it will see; the incident view weights each fragment by one over its incident's fragment count and shows how it performs across distinct situations. A model that looks strong only in the traffic view has mostly learned the big outages.
Run one influence check before trusting any pooled result: recompute the metric with each of the largest incidents removed, then with the top 1 percent removed together. If the conclusion flips, the result describes a few outages rather than the operation.
Keep every fragment of an incident on one side of the split
Assign whole incidents to training, validation and test, never individual records. scikit-learn's cross-validation guide (repository copy, checked 2026-10-27) says grouped data breaks the independence assumption and that a fair test draws every validation sample from groups absent from the paired training fold; its GroupKFold splitter keeps each group out of one side or the other.
Incident data adds two leak paths that a row-level shuffle misses. Copied resolution text, such as a problem ticket's closing comment pushed into every linked ticket, puts the answer for a test ticket into training under a different ID. Follow-up records, the postmortem and its action items, describe the outcome in plain words, so a model trained on them has already read the test incident's ending.
Add a time boundary on top of the incident boundary when the model will run on future incidents. Train on earlier months, test on later ones, and keep recurrence chains together so a fault that returns in the test period was not resolved in training under its first incident number.
Audit the split after assigning it. Search the test records for incident keys, problem IDs, change references and long resolution strings that also appear in training; every hit is a leak to fix before anything is scored, and the hit count belongs in the dataset card.
Could a hundred tickets from one outage really be a hundred tasks?
Sometimes, and the test is whether each fragment carries its own decision and its own success check. During a billing outage, each affected customer may need a separate credit decision with its own eligibility check and reply; those are distinct tasks that happen to share a cause.
Even then the fragments are not independent for statistics. They share the cause, the policy in force that day and often one agent's phrasing, so they count as tasks for coverage but stay clustered for intervals and splits. Report them as 100 credit decisions from 1 incident, not as 100 independent cases.
The opposite error also happens. An aggressive linking rule can fold two separate faults into one incident because they hit the same service in the same hour, which hides real variety. Review a sample of merges as well as links, and give reviewers an override field to split an incident the heuristic joined.
A large export can also hide incidents that never produced a record, such as a fault one engineer caught and fixed without paging anyone. Those are missing from both counts. No roll-up can recover them, so name the gap in the limitations rather than implying the archive saw every failure.
Coverage limitations an incident count still hides
Rolling up to incidents fixes the count, not the coverage. The table lists the limitations most likely to remain after the roll-up, how to check each one and the wording a coverage statement can honestly use.
| Limitation | Why it survives the roll-up | Check | Wording the evidence supports |
|---|---|---|---|
| Silent incidents | Faults fixed without an alert or ticket leave nothing to roll up | Compare change and deploy logs with incident records for unexplained rollbacks | Covers incidents that reached alerting or the help desk |
| Grouping drift | Alert grouping and ticket-linking settings changed during the window | Settings history, with counts reported per configuration period | Incident counts compare within, not across, the listed periods |
| Dominant outages | A few long incidents still supply a large share of fragments | Top 1 percent share and an influence check without them | Results shown with and without the largest incidents |
| Linking errors | Heuristic links join separate faults or miss true fragments | Two-person review of sampled links and merges, with an error rate | Incident boundaries reviewed on a sample; error rate stated |
| Missing endings | Some incidents have no recorded recovery or cause | Share of incidents with recovery evidence and a stated cause | Outcome claims limited to incidents with recovery evidence |
| Service concentration | One service or team may produce most incidents | Incidents per service against the services in scope | Coverage stated per service |
Writing the incident unit into a data request
State the unit and the count you need in incidents, then the fragments you want inside each. A request for 10,000 tickets invites an export dominated by a few outages; a request for a number of incidents across named services, with fragments attached and linking fields intact, yields a sample a reviewer can count.
SourceX does not keep datasets in stock; it sources them against a buyer's written requirement covering scope, volume, language, format, timing and licensing needs. Any candidate incident archive still has to pass rights, privacy and quality review, so a request implies nothing about what exists or what will be accepted.
For incident records, five additions to the request give the supplier and the reviewer the same count to check:
- Decision and unit: the task the model will perform and what counts as one task.
- Incident count by service and severity band, with the export window.
- Linking fields kept intact: parent, problem, grouping, merge and change references.
- A per-incident fragment cap and the rule for choosing which fragments to keep.
- Incident-level splits and an acceptance test that reports incident counts beside record counts.
Frequently asked questions
What is incident fragmentation in an AI dataset?
Incident fragmentation is when one real incident appears as many records, such as repeated alerts, duplicate customer tickets, chat threads, status updates and follow-up tasks. Each record has its own ID, so exports count them separately. For sampling they are correlated pieces of one case and should be counted, split and analyzed together.
Should duplicate customer tickets from one outage be deleted?
Usually not. They hold real variety in how customers describe a problem and how agents reply, which some tasks need. Keep them, attach them to the parent incident, cap how many enter a fragment-level sample and keep all of them on the same side of any split.
How do you split incident data into training and test sets?
Assign whole incidents, not records, to each split, and add a time cutoff when the model will face future incidents. Keep recurrence chains and follow-up records such as postmortems with the incident they describe. Group-aware splitters such as scikit-learn's GroupKFold apply the rule once every record carries an incident key.
Does alert grouping in the monitoring tool already solve fragmentation?
Only partly. Grouping depends on settings such as time windows, incident age limits and similarity models, so one long outage can still become several incidents, and settings may change during the export window. Treat the tool's incident ID as one linking source, add help desk and change links, and review a sample of the result.
How many incidents does an evaluation need?
That depends on the decision, the smallest difference that matters and how varied incidents are, so size the set with incidents as the unit rather than records. Then check that each service and severity band you care about has enough incidents on its own. A large fragment count cannot stand in for missing incidents.
Sources
- Anthropic defines an evaluation task as a single test with defined inputs and success criteria, and the outcome as the environment's final state at the end of a trial. Source
- Anthropic recommends clustering standard errors on the sampled unit when evaluation items are not independent. Source
- PagerDuty groups a new alert into an existing incident only inside the time window, for incidents under 24 hours old and for alerts its model judges similar. Source
- In Zendesk, solving a problem ticket solves its linked incident tickets and adds the problem's closing comment to those not already solved. Source
- The Cochrane Handbook treats analyzing clustered data as independent as a unit-of-analysis error that makes intervals too narrow and P values too small. Source
- scikit-learn's guide says grouped data breaks the independence assumption and that GroupKFold keeps a group out of both the training and test sets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.