Skip to content

Systems and records

How to test that status page and support replies tell one outage story

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Test IT outage communication consistency by rebuilding the incident timeline as the reference, then pairing every status page update and support reply with the incident state at its send time. Score each pair on phase, affected service, impact, region, workaround, timing promises and resolution, label it consistent, lagging, contradictory or over-disclosed, and record approved exceptions separately.

Record timeline

IT outage communication

This page covers how to test that status page and support replies tell one outage story.Illustrative

Key takeaways

  • Judge both channels against the incident record, not against each other, or a late status page becomes the standard.
  • Store the exact text sent and the minute it was sent; current macros and edited status posts are not evidence.
  • Incident facts must match across channels, while account-specific detail may legitimately differ.
  • Set a lag tolerance before scoring and measure status cadence separately from support lag.
  • Remove credentials, internal hostnames and customer identities before incident histories leave the company.

How do you test consistency between a status page and customer support replies?

Rebuild the incident's own timeline first, then pair every outbound message, public status update or support reply, with the incident state at the minute it was sent, and score each pair on a fixed set of dimensions: phase, affected service, customer impact, region, workaround, timing promises and resolution. A reply is consistent when it says nothing the status page contradicts at that moment and nothing the incident record did not yet know.

Testing against each other is the trap. If you compare support replies only with the status page, a status page that was itself late or wrong becomes the standard, and a support agent who correctly warned customers ten minutes earlier is scored as inconsistent. The incident record, with its alert evidence and approval trail, is the reference; the two public channels are both measured against it.

The result is a cross-channel incident consistency benchmark: a set of dated message pairs with expected labels, which you can use to audit past incidents, to train or evaluate an AI assistant that drafts outage replies, or simply to show a support team where its macros fall behind.

Why the status page and the support queue drift apart during an outage

They drift because different people write them on different clocks from different information. Status updates usually pass through an approval step tied to the incident lead, while support replies are written ticket by ticket, often from macros prepared before anyone knew this incident existed.

PagerDuty's public incident response guide shows why the approval step exists. Its Customer Liaison drafts the public message, gets sign-off from the Incident Commander and only then posts, because the nature of the incident may have changed while the message was being written. The same guide asks the liaison to pass customer reports back to the Incident Commander, since the volume of calls is evidence about scope.

Help desk mechanics add their own lag. Zendesk documents a problem and incident ticket model in which customer reports are set as incidents linked to one problem ticket. Solving the problem solves the linked incidents, a comment added at that moment goes only to linked incidents that are still unsolved, and reopening the problem does not reopen incidents already solved. Each of those rules decides which customers heard the final message and which did not, so the benchmark has to read them from the ticket history rather than assume a broadcast reached everyone.

What is the source of truth for each minute of an incident?

The incident record is the reference, and the benchmark should point to the specific entry that established each fact. Google's SRE book describes the incident commander keeping a living incident document and a communications lead issuing periodic updates to responders and stakeholders. That document, plus alert history and the approval trail for public messages, tells you what the response team knew and when.

Status page tools keep two separate kinds of state that are easy to conflate. Atlassian Statuspage moves an incident through Investigating, Identified, Monitoring and Resolved, while each component carries its own status, such as Operational, Degraded Performance, Partial Outage, Major Outage or Under Maintenance. A page can say Monitoring while a component still shows Partial Outage, so the benchmark records both.

What is the source of truth for each minute of an incident?
RecordTypical systemWhat it establishesCaveat when building the benchmark
Alert and paging historyPagerDuty, Opsgenie, monitoring toolsWhen symptoms were detected and who was engagedDetection time is not the time impact began
Incident document or channel logWiki, Google Docs, Slack incident channelWhat responders believed and decided, minute by minuteContains internal detail that must be redacted
Public status historyStatuspage or similarIncident phase, component status and wording at each postEdits to earlier posts may overwrite the original text
Message approval trailChat sign-off or ticket commentWho approved each public update and whenOften informal; mark missing approvals explicitly
Support tickets and repliesZendesk, Jira Service Management, FreshdeskWhat each customer was told and whenMacro text versions change; store the text actually sent
PostmortemPostmortem document or trackerFinal impact, duration and data loss statementWritten after the fact; never use it to judge what was knowable earlier

The cross-channel incident consistency benchmark

One benchmark item is one outbound message judged against the incident state at its send time. Build items for every status post and for every support reply on a ticket linked to the incident, plus a sample of replies on unlinked tickets that arrived during the incident window, because those are where 'no known issues' answers hide. The version shown here is v1, dated 2026-10-18.

At a glance

The cross-channel incident consistency benchmark

The cross-channel incident consistency benchmark
FieldWhat it holdsValidation check
incident_refPseudonymous incident id and severity at that minuteSeverity taken from the incident record, not the postmortem
message_ref and channelMessage id; status page, support reply, social or account emailExact sent text stored, not the current macro
sent_atSend time with time zone, plus minutes since incident startFalls inside the incident window or the stated follow-up period
state_at_sendPhase, affected components, impact, regions, workaround known at that minuteEach value links an incident document or alert entry
claims_in_messageThe message broken into the same dimensions as state_at_sendEvery dimension filled or marked not mentioned
approvalApprover role and time, or nonePublic posts without approval flagged
label_per_dimensionconsistent, lagging, ahead_unconfirmed, contradictory, over_disclosedLagging requires a tolerance window stated in the item
overall_labelWorst dimension label, plus one-sentence rationaleRationale cites the incident entry that decides it
exception_flagIntentional divergence with reason and approverOnly allowed with a named approver role

Which details must match, and which may legitimately differ?

Facts about the incident must match; detail about one customer's account may differ. A support reply can tell a customer that their workspace is in the affected region, which the status page will never say, but it cannot describe a different phase, a different set of affected services or a different promise about timing.

PagerDuty's guidance gives a useful default for two of the rows. It tells the liaison never to estimate resolution times and not to provide too much detail, such as naming a specific server, because customers care about the effect on them. It also says customers on premium support arrangements may receive impact communication individually, which is a legitimate source of extra detail in one channel.

Which details must match, and which may legitimately differ?
DimensionStatus pageSupport replyConsistency rule
PhaseInvestigating, Identified, Monitoring, ResolvedPlain-language equivalentNever earlier or later than the incident record at send time
Affected service and regionComponent names and regionsMay add whether this customer is affectedNo component the status page omits without an incident entry
Customer impactWhat users experienceSame symptom, possibly for this accountSame severity wording; no softer or harsher description
WorkaroundPublished when knownMay walk the customer through itOnly workarounds the incident record approved
TimingNext update time, no fix estimateNext update time or link to the pageNo resolution estimate in either channel
Resolution and data lossFull recovery and any data loss statedSame statement, plus account follow-upResolution claims only after recovery is confirmed
Technical detailEffect-level descriptionEffect-level descriptionNo hostnames, internal paths or security detail

Timing labels: how late is too late for a support reply?

Set a tolerance window before scoring, write it into each item, and score lag against it instead of treating any delay as a failure. A reply sent two minutes after a status change may reasonably still describe the previous phase; a reply sent an hour after resolution that still says the team is investigating is stale.

PagerDuty's published external communication guidelines are one reasonable reference policy. They call for the initial message within 5 minutes of the incident call starting, a scoping message within 5 minutes after that, updates at least every 20 minutes during the first two hours or whenever impact changes meaningfully, a different cadence for incidents longer than two hours, and a final message only after full recovery is confirmed, stating any data loss. Your own policy may differ; the benchmark should test against the policy your team actually had at the time.

Score status cadence and reply lag separately. Cadence measures whether the public channel kept its promises; lag measures whether support followed the public channel. Mixing them hides which team needs a fix.

Report a small set of aggregate figures per incident and across incidents: the share of replies labeled contradictory, the median and worst support lag after each status change, the count of replies still describing an earlier phase after resolution, and the count of replies sent on unlinked tickets while the incident was open. Publish the item count and the number of excluded exception items next to every figure, so nobody reads a clean rate from a handful of messages as proof of a healthy process.

Worked example (illustrative): delayed notifications in one region

This incident is illustrative and does not describe any real company or customer. Notifications to customers in one region are delayed. Times are minutes since the incident call started, and the tolerance window for support lag is set at 10 minutes.

Worked example (illustrative): delayed notifications in one region
MinuteMessageState at send (from incident record)Label
4Status page: Investigating delayed notificationsSymptoms seen, cause unknownconsistent
7Support macro: no known issues, please clear your cacheIncident open for 7 minutescontradictory
9Status page: Identified, one region, Partial OutageQueue backlog confirmed in one regionconsistent
16Support reply: we expect a fix within 10 minutesFix not yet deployedcontradictory (timing promise)
24Support reply names an internal host and queueSame stateover_disclosed
48Status page: Resolved, no data lossBacklog drained, recovery confirmedconsistent
63Support reply: still investigatingResolved 15 minutes earlierlagging, outside tolerance

When different messages are correct: the intentional divergence exception

Some divergence is deliberate and right. During a security incident, public wording may be held back on the advice of security or legal staff while affected customers receive direct notices; one customer may be hit by a separate, account-specific fault during a platform incident; or a large customer may get an individual briefing under its support arrangement. Mark these items with exception_flag, the reason and the approver role, and exclude them from the contradiction rate.

Do not let the exception swallow the benchmark. If more than a small share of items carry the flag, the problem is usually that the incident record does not capture decisions about messaging, and that gap is the finding. Security incidents in particular belong with a security reviewer before any item is built from them.

What must be removed before incident histories leave the company?

Strip credentials, internal topology and anything that would help an attacker before an item enters a shared dataset. The OWASP Logging Cheat Sheet lists data that should not be recorded directly in logs, including access tokens, authentication passwords, database connection strings and encryption keys, and calls out file paths and internal network names and addresses as data needing special care. Incident channels and support threads routinely contain all of these.

Customer identities need the same care: replace names, email addresses and account ids with consistent pseudonyms so that the pairing of a ticket with an incident survives. Keep the effect-level wording intact, because it is exactly what the benchmark scores. An IT operations lead and a security reviewer should sign off on the redaction rules before the first export.

Licensing outage communication histories as AI data

Paired incident records and customer replies are useful to teams building support and operations assistants, because they show how a real organization communicated under uncertainty. They are only useful if the pairing, timestamps and approvals survive preparation.

An incident archive is handled like any other SourceX license, with rights review, contracting, preparation, delivery and payment run on the company's behalf. Qualification opens with a short assessment and no upload; delivery waits until the company has approved the scope, typically the systems, date range and redaction rules your security reviewer agreed, and signed.

Frequently asked questions

Should support agents link customers to the status page instead of describing the outage?

Linking is a good default because it keeps one public account of the incident, but the reply should still confirm in plain words whether the customer is affected. A bare link sent before the status page acknowledges the incident tells the customer nothing and can read as a denial.

Is it inconsistent if support replies were ahead of the status page?

Not necessarily. Score the reply against the incident record: if responders had already confirmed the fact and the reply was accurate, it is consistent; use ahead_unconfirmed only when the record had not yet confirmed it. Being early and right is a status page cadence problem, not a support problem.

Which tickets should be included in the benchmark?

Include every reply on tickets linked to the incident and a sample of replies on unlinked tickets created during the incident window. The unlinked sample is where agents unaware of the incident tell customers there is no known issue.

How do problem and incident tickets affect who received the final update?

In Zendesk, solving a problem ticket solves its linked incident tickets, and a comment added at that time goes only to linked incidents that are still unsolved. Customers whose tickets were closed earlier may never receive the resolution message, which the benchmark should record.

Can AI-drafted outage replies be evaluated with this benchmark?

Yes. Give the model the incident state at a chosen minute and a customer ticket, then score its draft on the same dimensions and labels. Keep a human approval step for public messages in production, as incident response guides recommend for status posts.

Sources

  • PagerDuty's incident response guide sets initial communication within 5 minutes, scoping within 5 minutes after, updates at least every 20 minutes for the first two hours, a long-incident model after two hours, a resolution message after confirmed recovery stating data loss, and individual communication for customers with special support contracts. Source
  • PagerDuty's Customer Liaison drafts public messages, gets Incident Commander approval before posting because the incident may have changed, relays customer reports, and avoids resolution estimates and excess technical detail. Source
  • Atlassian Statuspage incidents move through Investigating, Identified, Monitoring and Resolved, and components carry separate statuses such as Degraded Performance, Partial Outage and Major Outage. Source
  • In Zendesk, solving a problem ticket solves linked incident tickets, a comment at solve time goes only to unsolved linked incidents, and reopening a problem does not reopen solved incidents. Source
  • Google's SRE book describes the incident commander keeping a living incident document and a communications lead issuing periodic updates to the response team and stakeholders. Source
  • The OWASP Logging Cheat Sheet lists access tokens, passwords, database connection strings and encryption keys as data not to log directly, and file paths and internal network names as needing special treatment. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies