AI uses for records
Outage support needs its own evaluation set, keyed to the incident clock
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Treat outage support as its own task distribution. Once an incident is declared, the right reply to an affected customer is acknowledgement, scope and the next update, not device troubleshooting, while an unaffected customer still needs ordinary troubleshooting. Key every contact to the incident's state at send time and to its affected scope, then grade by window. None of tau2-bench's 2,285 telecom support tasks puts the fault on the network side.
Evaluation split
Outage support dataset
Key takeaways
- An outage changes the cause, the correct action and the evidence of success, so outage-period contacts need their own labels and their own scores.
- Each contact needs two keys: whether an incident was declared when the reply was sent, and whether this customer was inside its affected scope.
- The costliest errors sit in the overlaps: troubleshooting a customer the incident explains, and blaming the incident for a fault it does not explain.
- In tau2-bench's 2,285 telecom support tasks every injected fault sits on the phone, SIM, an app or the account; none is network-side, and the No Service script never checks for an outage.
- Ticket closure is weakest exactly here: in Zendesk, solving one problem ticket sets every linked incident ticket to solved.
- Hold out whole incidents when testing, and report each window's score separately.
Why is support during an outage a different task?
Because the cause, the right action and the proof of success all change at once. On an ordinary day a customer who cannot connect usually has something wrong on their side, and the agent's job is to find it. During a declared provider incident the cause is shared and already known, the agent cannot fix it, and the right reply tells the customer what is affected, what to do in the meantime and when the next update will come.
Regulators and cloud providers draw the same line in their own records. The FCC's outage reporting rules define an outage as "a significant degradation in the ability of an end user to establish and maintain a channel of communications" resulting from failure or degraded performance in the provider's network, which puts the fault outside the customer's device. AWS Health labels each event as public or account-specific, and Microsoft's Resource Health FAQ explains that when a host reboots unexpectedly, only customers whose virtual machines were running on that host are alerted.
A support evaluation set that pools outage-period contacts with ordinary ones scores two different jobs with one rubric. It rewards troubleshooting where acknowledgement was right, and it hides the cases where an agent blamed the incident for something the incident did not cause.
A count of 2,285 telecom support tasks finds no network-side fault
Public support benchmarks mostly model the ordinary day. The telecom domain of Sierra's tau2-bench simulates a customer who controls a phone while the agent guides the fix. In its task file at commit 9970d4b (October 27, 2026), each of the 2,285 tasks sets up its fault with initialization actions drawn from 20 functions, and apart from scenario setup every one acts on the phone's settings, the SIM, an app permission or the customer's account and line. None represents a problem in the carrier's network, and the word outage appears nowhere in the domain's policy, manual, workflow or code.
The troubleshooting script follows from that. For a customer with No Service, the workflow checks airplane mode, then the SIM, then has the customer reset APN settings and restart the phone, then checks for a suspended line, then escalates. During a real carrier outage the same script would walk an affected customer through a settings reset and a restart that cannot help.
This is not a defect in tau2-bench, which was built to test agents that guide a user through device fixes. It does mean a team cannot borrow outage evaluation from it, and should expect the same gap in its own historical tickets unless incident links were recorded. The table groups the fault-setting functions by where the injected condition sits:
| Where the injected condition sits | Tasks touching it (of 2,285) | Base set (of 114) | Example functions |
|---|---|---|---|
| Phone settings | 2,261 | 108 | turn_airplane_mode_on, break_apn_settings |
| Customer account or line | 1,992 | 97 | suspend_line_for_overdue_bill, set_data_usage |
| App permission | 1,536 | 35 | remove_app_permission |
| SIM card | 1,028 | 50 | unseat_sim_card, lock_sim_card |
| Carrier network | 0 | 0 | None in the task files |
Four kinds of contact during an incident, plus one after it
Two questions sort every incident-period contact: had the provider declared the incident when the reply went out, and was this customer inside the affected scope? Answer both from records, not from the wording of the reply, and each contact lands in one of four cells, with a fifth kind after resolution.
The overlaps carry most of the risk. A contact inside the scope of a declared incident should get the incident message, not a device checklist. A contact outside the scope during the same hour, the outage shadow, still needs ordinary troubleshooting, and blaming the incident is a failure even when the customer accepts the explanation. Contacts before declaration cannot be graded on knowledge the agent did not have, but they can be graded on whether a matching pattern was escalated. After resolution, an affected customer who is still broken is an ordinary case again, with the incident as history. The table states the correct action and the record that proves each label:
| Contact type | How to identify it in records | Correct action | Typical failure |
|---|---|---|---|
| Declared and in scope | Incident open at send time; customer matches the affected product, region or resource | Acknowledge, state scope and workaround, give the next update time | Device steps or an invented restoration time |
| Declared, out of scope (outage shadow) | Incident open; customer outside the affected scope | Troubleshoot as on an ordinary day | Blaming the incident and closing the case |
| Before declaration, later in scope | Contact precedes declaration; customer inside the scope declared later | Brief triage, then report the pattern | Long device scripts and no escalation |
| Unrelated | No incident open, or one that cannot explain the symptom | Ordinary troubleshooting | None specific to outages |
| After resolution, still affected | Incident resolved; same customer reports the same symptom | Verify restoration, then troubleshoot lingering effects | Assuming the fix reached everyone |
Why ticket closure and satisfaction mislead most during an outage
Because during an incident they mostly record the provider's fix, not the agent's reply. Zendesk's documentation describes the common pattern: reports of one problem become incident tickets linked to a problem ticket, and solving the problem ticket sets all the linked incident tickets to solved. Reopening the problem ticket does not reopen them; they stay solved.
A closure-based label therefore gives every agent who touched an incident ticket the same success, whether the reply was accurate, misleading or a wasted troubleshooting script. Satisfaction can swing with the outage itself, and a customer told that the outage explains everything may rate the contact well even when, in the outage shadow, that answer was wrong.
That makes the strongest alternative reading of any good outage score, that closure and sentiment are standing in for resolution, likely rather than hypothetical. The fix is to grade the reply against the incident record as it stood at send time, and to grade resolution against evidence that service returned for that customer, such as a resource-level health status or no repeat contact on the same symptom within a fixed window.
The as-of decision packet: one record per contact
One packet per contact holds everything a grader needs to place the contact in its window and check the reply. The as-of fields are the core: they record the incident and its status text as they stood when the reply was sent, not as they read after the postmortem.
At a glance
The as-of decision packet: one record per contact
An Illustrative packet: a customer writes at 14:07 UTC that card payments fail at checkout. The incident record shows a payments incident declared at 13:52 for one region, state investigating, with the last public update at 14:00 promising another by 14:20. The customer's account sits in that region, so the contact is declared and in scope. The reply acknowledges the incident, names the affected payment method, points to the documented alternative and says the next update is due by 14:20; it does not ask the customer to clear a browser cache. Service for that account returns at 15:31 by the resource status, and the customer does not write again within seven days. Every value is invented for illustration. The packet's fields:
- Contact: UTC timestamp, channel, the symptom in the customer's words, and pseudonymized scope keys such as product, plan, region and resource or account identifiers.
- Incident as of send time: incident identifier, declaration time, state (investigating, identified, monitoring or resolved), the affected-scope definition in force, and the public status text with its timestamp.
- Scope match: whether the customer's keys fall inside that scope, who decided, and from which record.
- Agent actions: each step and tool call with timestamps, including any device or account change the customer was asked to make.
- Reply: the sent text, the policy or template version in force, and any promise it made about timing or credits.
- Outcome evidence: when service returned for this customer and from which record, plus any repeat contact on the same symptom within the fixed window.
- Label: contact type from the five-row table, grader identity, rubric version and any disagreement note.
A grading rubric that scores each window on its own
Score the reply within its window, then report the windows separately, because a pooled outage score hides the shadow errors. Several rubric items borrow from incident communication practice that is already written down. PagerDuty's public incident response documentation says the first external message should show the incident is under investigation, a scoping update should name the impacted components and regions, updates should come at least every 20 minutes during the first two hours, and the resolution message should confirm full recovery and state any lingering impact.
The same documentation asks its Customer Liaison to track incoming customer support requests and how far the incident has progressed, from still investigating to close to resolution. That is the as-of join a grader needs, performed live. The rubric items assume the packet above:
| Rubric item | Applies to | Pass | Fail |
|---|---|---|---|
| Scope check performed | Every incident-period contact | Reply or notes show the scope match was checked | Scope assumed from the symptom alone |
| Incident acknowledged | Declared and in scope | Names the incident, the affected service and the next update | Silence about a declared incident |
| No futile device steps | Declared and in scope | No resets or reinstalls requested | Customer walked through a fix that cannot help |
| No false attribution | Outage shadow | Ordinary troubleshooting goes ahead | Symptom blamed on an incident that cannot explain it |
| Timing matches the record | Declared and in scope | Any time given matches the status text at send time | A restoration time invented or out of date |
| Pattern escalated | Before declaration | Matching symptoms reported to the incident channel | Several similar contacts and no report |
| Restoration verified | After resolution | Service confirmed for this customer before closing | Closed because the incident closed |
How to test on held-out incidents rather than held-out tickets
Hold out whole incidents. Contacts from one incident share a cause, a status text and often a macro, so a split by ticket lets a model learn the incident's wording from training contacts and repeat it on test contacts from the same hour. Split by incident identifier, keep every contact of an incident on one side, and reserve at least one incident type, such as a regional payments failure, that the training side never saw.
Test in two stages. First, measure whether the system assigns the contact type correctly from records alone, because every later score depends on it. Then grade replies within each true contact type and report, with denominators: acknowledgement accuracy for declared in-scope contacts, the false attribution rate in the outage shadow, the escalation rate before declaration, and verified restoration after resolution. A single pooled outage score should not be on the list.
Limits of incident-keyed evaluation
The design depends on records many companies keep unevenly, and these gaps decide how far a result can be trusted:
- Affected scope often changes during an incident; the packet has to keep each version of the scope with its timestamp, or early replies get graded against later knowledge.
- Incident records can lag the queue, so some contacts look pre-declaration only because the record was filed late; compare the declaration time with the first internal alert where both exist.
- Status text and support replies can disagree with each other; testing that consistency is a separate job with its own method.
- Per-customer restoration evidence exists for cloud resources and network lines more often than for software features, so the outcome field will be empty for some products.
- Incident records can carry security detail and customer identifiers; remove secrets and pseudonymize scope keys before any record leaves the company.
What to request when buying outage-period support records
Ask for incident-linked contacts, not a sample of tickets that happen to fall in outage hours. SourceX assembles datasets for a stated buyer need instead of selling from stock, and the request form is where a buyer describes scope, volumes, language, format, timing and licensing terms; for this task the scope should name the five contact types, the as-of incident fields and the outcome evidence. The company holding the records approves that scope and signs before anything is delivered.
Set an acceptance test in the same request: the share of contacts with a resolvable scope match, the share carrying as-of incident state and the share with outcome evidence, measured on a sample before the full delivery. Nothing here describes such a dataset as available; whether a given supplier's records can pass the test is settled case by case.
Methods note: how the telecom tasks were counted
The telecom figures come from github.com/sierra-research/tau2-bench at commit 9970d4b (dated October 27, 2026), checked the same day, with files read as data and no code run. Tasks are the 2,285 entries in data/tau2/domains/telecom/tasks.json; the base set is the 114 task ids listed under base in split_tasks.json. Each initialization action was assigned a location by its function name using a fixed table: phone settings, SIM, app permission, customer account or line, or carrier network. Scenario setup (set_user_info and set_user_location) and simulate_network_search, a refresh of the phone's connection state that 1,152 tasks call, are left out of the table.
A task counts once in every row it touches, so the rows sum to more than 2,285. A task would count as network-side if any of its actions sat in the carrier network row; no function in the task files belongs there, and no function went unclassified. The outage count is a case-insensitive search for the word in every Markdown file of the telecom domain folder, including the solo variants, and in the domain's Python source. Not measured: no agent was run, and no real support or incident records were used; the packet example is Illustrative.
Frequently asked questions
Should outage-period tickets be removed from a support evaluation set?
No; label them instead. Removing them hides the outage shadow, where ordinary faults arrive during an incident, and leaves a system untested in the hours when contact volume and risk peak. Keep them with incident keys and score them by window.
How can you tell whether a customer was affected by an incident?
Compare the customer's product, region, plan or resource identifiers with the incident's affected scope as it stood at the time of contact. Providers already record the distinction, for example AWS Health's public versus account-specific events, so the evidence often exists in operational systems.
What should a support reply say during a declared outage?
It should confirm the incident, say what is affected, give any documented workaround and state when the next update is due, matching the status text in force. It should not ask an affected customer to troubleshoot a device or promise a restoration time the incident record does not support.
Why not grade outage support by ticket closure or satisfaction scores?
Closure during an incident often reflects the parent fix: in Zendesk, solving a problem ticket marks every linked incident ticket solved. Satisfaction can swing with the outage itself. Neither shows whether the reply was right for that customer at that moment.
Can public support benchmarks test outage handling?
Not those built around device troubleshooting. In tau2-bench's telecom domain, all 2,285 tasks place the fault on the phone, the SIM, an app or the account, so outage handling needs incident-linked records or purpose-built scenarios.
Sources
- tau2-bench's telecom domain at commit 9970d4b (October 27, 2026): 2,285 tasks whose 20 initialization functions place every fault on the phone, SIM, app permission or account and line; no network-side condition and no use of the word outage; the No Service workflow checks airplane mode, SIM, APN reset and restart, then line suspension (recounted for this article). Source
- 47 CFR 4.5(a) defines an outage as a significant degradation in the ability of an end user to establish and maintain a channel of communications as a result of failure or degradation in the performance of a communications provider's network. Source
- AWS Health distinguishes account-specific events, which affect an account and name affected resources, from public events, which are not specific to any account. Source
- Microsoft's Azure Resource Health FAQ: Resource Health covers issues affecting a customer's specific resources; when a host reboots unexpectedly, only customers whose virtual machines ran on that host are alerted. Source
- Zendesk: incident tickets linked to a problem ticket are set to solved when the problem ticket is solved; reopening the problem ticket does not reopen them. Source
- PagerDuty incident response documentation: first external message indicates investigation; a scoping update names impacted components and regions; updates at least every 20 minutes during the first two hours; resolution confirms full recovery and lingering impact. Source
- PagerDuty's Customer Liaison role tracks incoming customer support requests and how far the incident has progressed. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.