AI uses for records
Support macro drift dataset: finding replies a policy change made wrong
By SourceX Editorial · Updated
9 sections
Short answer
Support data identifies macros a policy change invalidated by joining three dated records per ticket: the policy version that governed the customer, the macro revision the agent applied, and the case outcome after it matured. A send is stale only when its promise conflicts with that governing policy, so customers whose orders predate the change, and agents who edited the macro before sending, need their own labels.
Agent run
Support macro drift dataset
Key takeaways
- Macro drift is a conflict between a sent promise and the policy that governed that customer, not wording that merely sounds dated.
- Judge each send against three dated records: the policy version, the macro revision live at send time, and the matured case outcome.
- Key the policy lookup on the date that decides which terms apply, often the order or contract date, so grandfathered customers are not mislabeled.
- Judge the words the customer received, because agents sometimes fix an outdated macro before sending.
- Label an outcome resolved only after the ticket has closed and no reopen, follow-up or later correction points back to it.
How should support data identify macros a policy change invalidated?
Support data identifies an invalidated macro by lining up three dated records for every ticket: the policy version in force for that customer, the macro revision the agent applied, and what happened to the case afterward. A macro send is stale when the promise it carries, such as a return window, a fee waiver or an eligibility rule, conflicts with the policy that governed that customer on that date.
That definition matters because macro drift rarely looks like an error inside a single ticket. The reply is polite, the macro was approved, the agent followed process, and the ticket was solved. The problem only shows up when you compare the reply with a policy document that changed weeks earlier, or when the customer comes back because the promised outcome never arrived.
A dataset built for this question is therefore a set of complete cases rather than a pile of replies. Each case keeps the conversation, the macro identity and revision, the policy version, any human correction, reopenings and follow-ups, and a resolution label that was assigned only after the case had time to mature.
Which dated records have to line up before a send can be judged?
You need a policy timeline, a macro timeline and a ticket timeline, each with its own clock. Most help desks store the third well, the second partially, and the first not at all unless someone owns the knowledge base carefully. The table below uses Zendesk as the worked system because its documentation describes each record; other help desks have close equivalents under different names.
The gap to plan for is the macro timeline. The Macros API exposes a read-only updated_at value for the last change and an active flag, which tells you when a macro last moved but not what it said before. The account audit log does record changes to business rules, a category that covers macros, with a text description of each change; test in your own account whether that description keeps the earlier wording, and if it does not, start taking dated snapshots of macro bodies before you need them.
| Timeline | Record in Zendesk | What it gives you | What it does not give you |
|---|---|---|---|
| Policy version | Help center article revisions | A stored version each time a policy article is saved, which can be viewed or restored | The date a change took effect for customers, or which orders it applied to |
| Macro definition | Macros API: updated_at and active | When the macro last changed and whether agents can still see it | The text and actions the macro carried before the last edit |
| Macro change history | Account audit log, rule changes | A dated entry with a change description whenever a macro or other business rule changes | A dependable copy of the prior body; confirm what your account records |
| Macro use on a ticket | Ticket audit macro reference event | The macro ID and title applied in an update, plus whether the macro has since been deleted | Which revision of the macro was live at that moment, unless you join on dates |
| Case outcome | Ticket metrics: reopens and solved_at | How many times the ticket went from solved back to open or pending, and when it was solved | Whether the customer came back on a new ticket after closure |
| Return after closure | Follow-up tickets | A new ticket that references a closed one when the customer replies | A link from the original ticket's metrics; you have to join follow-ups yourself |
What counts as drift, and what only looks like it?
Treat drift as a mismatch between a specific promise in the sent text, or a specific action the macro performed, and the policy that governed the case. Wording that merely sounds dated is a style problem, not drift, and labeling it as drift will teach a model to flag harmless replies.
At a glance
What counts as drift, and what only looks like it
The most important look-alike is a grandfathered customer. When a policy changes, many businesses keep the old terms for orders, contracts or claims that started before the change date. A macro that quotes the old return window can be exactly right for an order placed before the cutover and wrong for one placed after. The label must therefore key on the date that decides which policy applies, which is often the order or contract date rather than the date of the conversation.
- Stale term: the reply states a window, limit, fee waiver or eligibility rule the governing policy no longer offers.
- Stale action: the macro sets a field, tag or status that routes the case under an old process, even when the visible text is fine.
- Stale reference: the reply links to or names a help article, form or team that the change retired or renamed.
- Scope drift: a macro written for one case type is applied to another that the policy treats differently.
- Edited at send: the agent changed the macro text before sending, so the macro revision and the sent words disagree; judge the sent words.
- Not drift: an older term quoted to a customer whose order or contract predates the change, if the policy grandfathers it.
What does the policy and response freshness evaluation task contain?
The task asks one question per case: given the policy that governed this customer, was the reply's promise current, stale, legitimately grandfathered, or impossible to judge from the evidence? Each row below is a field the task record carries, with the decision it supports. A team can copy the table into a schema as it stands.
Keep the governing policy text inside each task rather than pointing at a live page. Public agent benchmarks do the same: the tau-bench airline policy opens by fixing the current time the agent must assume, so every judgment is made against one dated rulebook. A freshness task needs that discipline twice, because the case has its own decisive date and the policy has its own effective date.
| Field | Content | Decision it supports |
|---|---|---|
| case_id and thread links | Original ticket, merged tickets and any follow-up tickets that reference it | Whether the case is complete enough to judge |
| decisive_date and its basis | The date that decides which policy applies, and whether it came from the order, contract or contact | Which policy version governs the case |
| policy_version_text | The full governing policy passage with its version label and effective date | What the reply was allowed to promise |
| macro_id and macro_revision | The macro applied and the revision live at send time, joined by date | Whether drift came from the template or the agent |
| sent_text | The words the customer actually received, after any edit | What was promised, judged on the real message |
| extracted_promise | One normalized claim, for example a window in days or an eligibility condition | A comparable unit for scoring |
| freshness_label | current, stale, grandfathered or undeterminable, with the reviewer's reason | The ground truth for the task |
| correction_events | Later internal notes, supervisor edits, refunds or apologies tied to the promise | Whether the business itself treated the reply as wrong |
| outcome_evidence | Reopen count, follow-ups and final resolution after the maturity window | Whether the problem stayed resolved |
How should a model or agent be scored on this task?
Score the judgment and the repair separately. A model can correctly spot a stale promise and still propose a fix that breaks a different rule, and a benchmark that blends the two hides which skill is missing.
tau-bench offers a useful pattern for the repair half. Its reward can require that the database end state match a target produced by replaying the expected actions, and that specified information appear in the agent's messages to the user. Applied to support, the end state is what the ticket should look like after a correct reply, such as the right fields and tags, and the required information is the current term stated plainly. Any path that reaches the right state and says the right thing passes.
| Outcome | What the model did | Score treatment |
|---|---|---|
| Correct flag and fix | Labeled the send stale and drafted a reply with the governing term and correct routing | Full credit |
| Correct grandfathering | Recognized the older term still applied because of the decisive date | Full credit; this is the case most models miss |
| Correct flag, wrong fix | Saw the conflict but quoted a different outdated term or set the wrong field | Partial credit on detection, none on repair |
| False alarm | Flagged a current or grandfathered promise as stale | Penalize; it erodes trust in the reviewer queue |
| Missed drift | Passed a stale promise as current | Penalize most heavily when the case later reopened |
| Honest abstention | Marked the case undeterminable because the decisive date or policy text was missing | Credit when the evidence really is missing |
Which signals show the customer's problem did not come back after closure?
A solved status is a claim, not an outcome. In Zendesk a solved ticket can be reopened by the customer until it closes, and the reopens metric counts each time a ticket went from solved back to open or pending. Once a ticket is closed it cannot be reopened, so a customer reply creates a new follow-up ticket that references the closed one instead.
That behavior sets the maturity window for your outcome label. Zendesk's default automation closes solved tickets after four days, admins can change that, and solved tickets close automatically 28 days after they are solved regardless of other rules. A case is only ready for an outcome label after it has closed and you have searched for follow-ups that point back to it, plus any further period your business uses for chargebacks or warranty claims.
Join three signals before you write resolved: no reopen, no follow-up referencing the case, and no later correction event such as a refund or apology tied to the same promise. When any one is missing because the data was not exported, label the outcome unknown rather than resolved.
What happens when a return window shrinks before the macro does? An illustrative case
All numbers in this example are illustrative. A retailer shortens its return window from 60 days to 30 days for orders placed on or after March 1. The policy article is revised on February 20. The returns macro is edited 12 days after the cutover, on March 13. In the gap, agents send the old macro on 340 tickets.
A naive drift label marks all 340 sends stale because they quote 60 days after the policy changed. The freshness task does better. Joining each ticket to its order date shows that 210 of those customers ordered before March 1, so the 60-day promise was correct for them and the label is grandfathered. The remaining 130 sends are stale.
Outcome evidence then separates harm from noise. Of the 130 stale sends, suppose 41 cases show a reopen or a follow-up ticket when the customer tried to return an item between day 31 and day 60, and 17 show a supervisor approving the return anyway. Those 17 are the most valuable rows in the set: the business corrected course, and the record shows both the stale promise and the human repair.
The exception worth keeping in the set is the agent who edited the macro at send. On a handful of tickets an agent replaced 60 with 30 before sending. Those replies are current even though the macro revision was stale, which is why the task judges sent_text rather than the template.
When does a freshness study need narrowing or a specialist?
Narrow the scope when you cannot establish the decisive date for a case type. If orders, contracts or claims are not linked to tickets, keep only the case types where they are and record the exclusion; a smaller set with defensible labels is worth more than a large one with guessed policy versions.
Bring in a specialist when the promise touches rights the business does not set alone, such as statutory refund or cancellation rights, warranty law or regulated financial or insurance terms. There the governing rule may be a law rather than a help article, and labeling a reply current or stale becomes a legal judgment.
Bring in a privacy reviewer before the cases leave the help desk. Ticket text carries names, addresses, order numbers and sometimes payment or health details in free text, and rendered macros often insert the customer's name and account details into the reply itself.
Freshness-labeled support history as a licensed dataset
A support team that already keeps macro snapshots, policy revisions and linked outcomes holds an unusually complete record of how a business handled change. Licensing that history for AI is work SourceX takes on end to end, from qualification and rights review to contracting, preparation, delivery and payment.
Eligibility for a support team's history is checked through a brief assessment that needs no ticket export. Tickets are delivered only after the company approves scope and terms and signs, and the de-identified dataset is then used or shared only as that agreement allows.
Frequently asked questions
What is support macro drift?
Support macro drift is the gap that opens when a saved reply keeps promising terms the business has changed. It shows up as replies quoting an old window, fee rule or process after the policy moved. It is measured by comparing each sent reply with the policy version that governed that customer.
Can I find stale macros from the Zendesk macros list alone?
Not reliably. The Macros API shows when a macro last changed and whether it is active, but not its earlier wording or which tickets received each version. You need ticket audit events for macro use, dated macro snapshots or audit log entries, and the policy revision history to judge individual sends.
Is an old policy term always a mistake?
No. Many businesses honor the earlier terms for orders or contracts that started before a change, so an older term can be correct for some customers. Key every label on the date that decides which policy applies, which is often the order date, not the conversation date.
How long should I wait before labeling a ticket resolved?
At least until the ticket has closed and you have checked for follow-up tickets that reference it. In Zendesk solved tickets can still be reopened until closure, and replies after closure create follow-ups. Add any longer window your business uses for returns, chargebacks or warranty claims.
Should edited macro sends be removed from the dataset?
Keep them and judge the words the customer received. An agent who corrected an outdated macro before sending produced a current reply and a useful example of human repair. Record both the macro revision and the sent text so the difference is visible.
Sources
- tau-bench evaluates customer service agents in domains that each specify a policy the agent must follow, a set of tools and a set of tasks. Source
- The tau-bench airline agent policy opens by stating the current time the agent must assume. Source
- tau-bench rewards can require the database end state to match a target produced by replaying expected actions, and required information strings to appear in the agent's messages. Source
- Zendesk macros carry a read-only updated_at value for the last change and an active flag that controls whether the macro is displayed. Source
- Zendesk's account audit log records changes with a change description, and its rule source type covers macros, triggers, automations and views. Source
- A Zendesk ticket audit can include a macro reference event that records the macro's ID, title and whether it has been deleted. Source
- Zendesk help center stores a new version each time an article is saved, and earlier versions can be viewed or restored. Source
- Zendesk ticket metrics include a reopens count and a solved_at timestamp for each ticket. Source
- A closed Zendesk ticket cannot be reopened; a customer reply creates a follow-up ticket that references the closed one. Source
- Zendesk's default automation closes solved tickets after four days, and solved tickets close automatically 28 days after being solved. Source
See if your company qualifies
A short company assessment. No data uploads are needed.