Skip to content

AI uses for records

Support macro drift dataset: finding replies a policy change made wrong

By SourceX Editorial · Updated

9 sections

Short answer

Support data identifies macros a policy change invalidated by joining three dated records per ticket: the policy version that governed the customer, the macro revision the agent applied, and the case outcome after it matured. A send is stale only when its promise conflicts with that governing policy, so customers whose orders predate the change, and agents who edited the macro before sending, need their own labels.

Agent run

Support macro drift dataset

This page covers support macro drift dataset: finding replies a policy change made wrong.Illustrative

Key takeaways

  • Macro drift is a conflict between a sent promise and the policy that governed that customer, not wording that merely sounds dated.
  • Judge each send against three dated records: the policy version, the macro revision live at send time, and the matured case outcome.
  • Key the policy lookup on the date that decides which terms apply, often the order or contract date, so grandfathered customers are not mislabeled.
  • Judge the words the customer received, because agents sometimes fix an outdated macro before sending.
  • Label an outcome resolved only after the ticket has closed and no reopen, follow-up or later correction points back to it.

How should support data identify macros a policy change invalidated?

Support data identifies an invalidated macro by lining up three dated records for every ticket: the policy version in force for that customer, the macro revision the agent applied, and what happened to the case afterward. A macro send is stale when the promise it carries, such as a return window, a fee waiver or an eligibility rule, conflicts with the policy that governed that customer on that date.

That definition matters because macro drift rarely looks like an error inside a single ticket. The reply is polite, the macro was approved, the agent followed process, and the ticket was solved. The problem only shows up when you compare the reply with a policy document that changed weeks earlier, or when the customer comes back because the promised outcome never arrived.

A dataset built for this question is therefore a set of complete cases rather than a pile of replies. Each case keeps the conversation, the macro identity and revision, the policy version, any human correction, reopenings and follow-ups, and a resolution label that was assigned only after the case had time to mature.

Which dated records have to line up before a send can be judged?

You need a policy timeline, a macro timeline and a ticket timeline, each with its own clock. Most help desks store the third well, the second partially, and the first not at all unless someone owns the knowledge base carefully. The table below uses Zendesk as the worked system because its documentation describes each record; other help desks have close equivalents under different names.

The gap to plan for is the macro timeline. The Macros API exposes a read-only updated_at value for the last change and an active flag, which tells you when a macro last moved but not what it said before. The account audit log does record changes to business rules, a category that covers macros, with a text description of each change; test in your own account whether that description keeps the earlier wording, and if it does not, start taking dated snapshots of macro bodies before you need them.

Which dated records have to line up before a send can be judged?
TimelineRecord in ZendeskWhat it gives youWhat it does not give you
Policy versionHelp center article revisionsA stored version each time a policy article is saved, which can be viewed or restoredThe date a change took effect for customers, or which orders it applied to
Macro definitionMacros API: updated_at and activeWhen the macro last changed and whether agents can still see itThe text and actions the macro carried before the last edit
Macro change historyAccount audit log, rule changesA dated entry with a change description whenever a macro or other business rule changesA dependable copy of the prior body; confirm what your account records
Macro use on a ticketTicket audit macro reference eventThe macro ID and title applied in an update, plus whether the macro has since been deletedWhich revision of the macro was live at that moment, unless you join on dates
Case outcomeTicket metrics: reopens and solved_atHow many times the ticket went from solved back to open or pending, and when it was solvedWhether the customer came back on a new ticket after closure
Return after closureFollow-up ticketsA new ticket that references a closed one when the customer repliesA link from the original ticket's metrics; you have to join follow-ups yourself

What counts as drift, and what only looks like it?

Treat drift as a mismatch between a specific promise in the sent text, or a specific action the macro performed, and the policy that governed the case. Wording that merely sounds dated is a style problem, not drift, and labeling it as drift will teach a model to flag harmless replies.

At a glance

What counts as drift, and what only looks like it

The most important look-alike is a grandfathered customer. When a policy changes, many businesses keep the old terms for orders, contracts or claims that started before the change date. A macro that quotes the old return window can be exactly right for an order placed before the cutover and wrong for one placed after. The label must therefore key on the date that decides which policy applies, which is often the order or contract date rather than the date of the conversation.

  • Stale term: the reply states a window, limit, fee waiver or eligibility rule the governing policy no longer offers.
  • Stale action: the macro sets a field, tag or status that routes the case under an old process, even when the visible text is fine.
  • Stale reference: the reply links to or names a help article, form or team that the change retired or renamed.
  • Scope drift: a macro written for one case type is applied to another that the policy treats differently.
  • Edited at send: the agent changed the macro text before sending, so the macro revision and the sent words disagree; judge the sent words.
  • Not drift: an older term quoted to a customer whose order or contract predates the change, if the policy grandfathers it.

What does the policy and response freshness evaluation task contain?

The task asks one question per case: given the policy that governed this customer, was the reply's promise current, stale, legitimately grandfathered, or impossible to judge from the evidence? Each row below is a field the task record carries, with the decision it supports. A team can copy the table into a schema as it stands.

Keep the governing policy text inside each task rather than pointing at a live page. Public agent benchmarks do the same: the tau-bench airline policy opens by fixing the current time the agent must assume, so every judgment is made against one dated rulebook. A freshness task needs that discipline twice, because the case has its own decisive date and the policy has its own effective date.

What does the policy and response freshness evaluation task contain?
FieldContentDecision it supports
case_id and thread linksOriginal ticket, merged tickets and any follow-up tickets that reference itWhether the case is complete enough to judge
decisive_date and its basisThe date that decides which policy applies, and whether it came from the order, contract or contactWhich policy version governs the case
policy_version_textThe full governing policy passage with its version label and effective dateWhat the reply was allowed to promise
macro_id and macro_revisionThe macro applied and the revision live at send time, joined by dateWhether drift came from the template or the agent
sent_textThe words the customer actually received, after any editWhat was promised, judged on the real message
extracted_promiseOne normalized claim, for example a window in days or an eligibility conditionA comparable unit for scoring
freshness_labelcurrent, stale, grandfathered or undeterminable, with the reviewer's reasonThe ground truth for the task
correction_eventsLater internal notes, supervisor edits, refunds or apologies tied to the promiseWhether the business itself treated the reply as wrong
outcome_evidenceReopen count, follow-ups and final resolution after the maturity windowWhether the problem stayed resolved

How should a model or agent be scored on this task?

Score the judgment and the repair separately. A model can correctly spot a stale promise and still propose a fix that breaks a different rule, and a benchmark that blends the two hides which skill is missing.

tau-bench offers a useful pattern for the repair half. Its reward can require that the database end state match a target produced by replaying the expected actions, and that specified information appear in the agent's messages to the user. Applied to support, the end state is what the ticket should look like after a correct reply, such as the right fields and tags, and the required information is the current term stated plainly. Any path that reaches the right state and says the right thing passes.

How should a model or agent be scored on this task?
OutcomeWhat the model didScore treatment
Correct flag and fixLabeled the send stale and drafted a reply with the governing term and correct routingFull credit
Correct grandfatheringRecognized the older term still applied because of the decisive dateFull credit; this is the case most models miss
Correct flag, wrong fixSaw the conflict but quoted a different outdated term or set the wrong fieldPartial credit on detection, none on repair
False alarmFlagged a current or grandfathered promise as stalePenalize; it erodes trust in the reviewer queue
Missed driftPassed a stale promise as currentPenalize most heavily when the case later reopened
Honest abstentionMarked the case undeterminable because the decisive date or policy text was missingCredit when the evidence really is missing

Which signals show the customer's problem did not come back after closure?

A solved status is a claim, not an outcome. In Zendesk a solved ticket can be reopened by the customer until it closes, and the reopens metric counts each time a ticket went from solved back to open or pending. Once a ticket is closed it cannot be reopened, so a customer reply creates a new follow-up ticket that references the closed one instead.

That behavior sets the maturity window for your outcome label. Zendesk's default automation closes solved tickets after four days, admins can change that, and solved tickets close automatically 28 days after they are solved regardless of other rules. A case is only ready for an outcome label after it has closed and you have searched for follow-ups that point back to it, plus any further period your business uses for chargebacks or warranty claims.

Join three signals before you write resolved: no reopen, no follow-up referencing the case, and no later correction event such as a refund or apology tied to the same promise. When any one is missing because the data was not exported, label the outcome unknown rather than resolved.

What happens when a return window shrinks before the macro does? An illustrative case

All numbers in this example are illustrative. A retailer shortens its return window from 60 days to 30 days for orders placed on or after March 1. The policy article is revised on February 20. The returns macro is edited 12 days after the cutover, on March 13. In the gap, agents send the old macro on 340 tickets.

A naive drift label marks all 340 sends stale because they quote 60 days after the policy changed. The freshness task does better. Joining each ticket to its order date shows that 210 of those customers ordered before March 1, so the 60-day promise was correct for them and the label is grandfathered. The remaining 130 sends are stale.

Outcome evidence then separates harm from noise. Of the 130 stale sends, suppose 41 cases show a reopen or a follow-up ticket when the customer tried to return an item between day 31 and day 60, and 17 show a supervisor approving the return anyway. Those 17 are the most valuable rows in the set: the business corrected course, and the record shows both the stale promise and the human repair.

The exception worth keeping in the set is the agent who edited the macro at send. On a handful of tickets an agent replaced 60 with 30 before sending. Those replies are current even though the macro revision was stale, which is why the task judges sent_text rather than the template.

When does a freshness study need narrowing or a specialist?

Narrow the scope when you cannot establish the decisive date for a case type. If orders, contracts or claims are not linked to tickets, keep only the case types where they are and record the exclusion; a smaller set with defensible labels is worth more than a large one with guessed policy versions.

Bring in a specialist when the promise touches rights the business does not set alone, such as statutory refund or cancellation rights, warranty law or regulated financial or insurance terms. There the governing rule may be a law rather than a help article, and labeling a reply current or stale becomes a legal judgment.

Bring in a privacy reviewer before the cases leave the help desk. Ticket text carries names, addresses, order numbers and sometimes payment or health details in free text, and rendered macros often insert the customer's name and account details into the reply itself.

Freshness-labeled support history as a licensed dataset

A support team that already keeps macro snapshots, policy revisions and linked outcomes holds an unusually complete record of how a business handled change. Licensing that history for AI is work SourceX takes on end to end, from qualification and rights review to contracting, preparation, delivery and payment.

Eligibility for a support team's history is checked through a brief assessment that needs no ticket export. Tickets are delivered only after the company approves scope and terms and signs, and the de-identified dataset is then used or shared only as that agreement allows.

Frequently asked questions

What is support macro drift?

Support macro drift is the gap that opens when a saved reply keeps promising terms the business has changed. It shows up as replies quoting an old window, fee rule or process after the policy moved. It is measured by comparing each sent reply with the policy version that governed that customer.

Can I find stale macros from the Zendesk macros list alone?

Not reliably. The Macros API shows when a macro last changed and whether it is active, but not its earlier wording or which tickets received each version. You need ticket audit events for macro use, dated macro snapshots or audit log entries, and the policy revision history to judge individual sends.

Is an old policy term always a mistake?

No. Many businesses honor the earlier terms for orders or contracts that started before a change, so an older term can be correct for some customers. Key every label on the date that decides which policy applies, which is often the order date, not the conversation date.

How long should I wait before labeling a ticket resolved?

At least until the ticket has closed and you have checked for follow-up tickets that reference it. In Zendesk solved tickets can still be reopened until closure, and replies after closure create follow-ups. Add any longer window your business uses for returns, chargebacks or warranty claims.

Should edited macro sends be removed from the dataset?

Keep them and judge the words the customer received. An agent who corrected an outdated macro before sending produced a current reply and a useful example of human repair. Record both the macro revision and the sent text so the difference is visible.

Sources

  • tau-bench evaluates customer service agents in domains that each specify a policy the agent must follow, a set of tools and a set of tasks. Source
  • The tau-bench airline agent policy opens by stating the current time the agent must assume. Source
  • tau-bench rewards can require the database end state to match a target produced by replaying expected actions, and required information strings to appear in the agent's messages. Source
  • Zendesk macros carry a read-only updated_at value for the last change and an active flag that controls whether the macro is displayed. Source
  • Zendesk's account audit log records changes with a change description, and its rule source type covers macros, triggers, automations and views. Source
  • A Zendesk ticket audit can include a macro reference event that records the macro's ID, title and whether it has been deleted. Source
  • Zendesk help center stores a new version each time an article is saved, and earlier versions can be viewed or restored. Source
  • Zendesk ticket metrics include a reopens count and a solved_at timestamp for each ticket. Source
  • A closed Zendesk ticket cannot be reopened; a customer reply creates a follow-up ticket that references the closed one. Source
  • Zendesk's default automation closes solved tickets after four days, and solved tickets close automatically 28 days after being solved. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies