Skip to content

AI uses for records

A support macro misuse dataset should grade the case, not the template

By SourceX Editorial · Updated

9 sections

Short answer

A support macro misuse dataset grades each macro send on two separate questions: was the template current and approved, and did this case meet the conditions that template assumes? Misuse is a yes to the first and a no to the second. Record each macro's preconditions as checkable facts, the case facts at send time and the end state afterward, then test on held-out macros; a solved status or a thank-you cannot show fit.

Grader rubric

Support macro misuse dataset

This page covers A support macro misuse dataset should grade the case, not the template.Illustrative

Key takeaways

  • Grade every macro send on two questions, template validity and case fit; misuse is a valid template sent to a case that fails one of its preconditions.
  • Macro actions write the tags and closed status an export later treats as labels, so those fields record the agent's choice, not the outcome.
  • In golang/vulndb at commit e0bad56, 946 of 4,710 reports (20.1%) name the functions that decide whether an advisory applies; the rest cannot narrow below a package or module.
  • A precondition card per macro revision, tied to the systems that confirm each fact, is what makes misuse labelable in the first place.
  • Hold out whole macros and a later period, and report per-macro denominators, undeterminable shares and grader agreement.

What counts as macro misuse when the macro itself is right?

Macro misuse is a send where the template was current, approved and accurate, yet the case in front of the agent did not meet a condition the template takes for granted. The wording passes review; the fit fails. A support macro misuse dataset exists to teach and test that second judgment, so it has to record the template's conditions and the case's facts side by side.

That definition separates misuse from two neighbors. A stale macro fails an earlier test, because its promise no longer matches policy; dated policy versions catch that, and it belongs in a drift dataset. A badly edited send fails on wording the agent added. Misuse is what remains when the template is fine and the choice was wrong: a password reset sent to an account that signs in through single sign-on, a refund-to-card macro for a card that has since been closed, a shipping-delay apology for an order that never shipped because it was canceled.

The decision here is narrow and practical. Can a buyer or a support team label each macro send as correct use, misuse, stale template or undeterminable, from records they can actually export? Three pieces of evidence settle it, all dated to the moment of the send: what the template assumed, what the case's own systems showed, and what state the account reached afterward.

Why the help desk record hides misuse by design

The record hides misuse because a macro writes its own outcome. Many help desk macros bundle actions with the text. Intercom's documentation, for example, lets a macro assign the conversation, add tags, snooze, close or reopen it, with or without a reply attached, and it runs the close step after the macro's other actions. Export that conversation later and the tag reads like a classification and the closed status like a resolution, yet both are the template's claim about the case, applied by the same click that sent the words.

Machine learning already has a name for labels produced this way. The Snorkel paper (Ratner and colleagues, 2017) treats heuristic labeling functions as noisy sources whose accuracies are unknown and must be estimated from where the sources agree and disagree, before their output trains anything. A macro's tags are a labeling function that an agent fires by hand. Treating them as ground truth assumes the agent always chose well, which is precisely what a misuse label is meant to test.

Clinical records show how far template text can sit from a case-specific judgment. In outpatient progress notes written in 2018 at one academic medical center, a mean of 29.4 percent of the text was typed directly and the remaining 70.6 percent was templated or copied, Rule and colleagues reported in JAMA Network Open (2021). A note full of correct template language says little about what the clinician checked that day, and a ticket full of correct macro language has the same weakness.

What 4,710 vulnerability reports show about recording applicability

Software security already separates a correct warning from a warning that applies, and its public records show how often the applicability condition goes unrecorded. In the Go vulnerability database, a report can list the vulnerable functions, which the format calls symbols. The repository's format guide puts the rule plainly: when symbols are included, only programs that use those symbols are marked vulnerable; when they are omitted, any program that imports the module is marked vulnerable.

We counted the report files in the golang/vulndb repository at commit e0bad56, committed October 20, 2026. A report counts as naming its condition when any of its packages lists symbols or derived symbols. Of 4,710 reports, 946 (20.1 percent) do; 74 stop at the package level and 3,690 (78.3 percent) name only a module and its affected versions. Review status accounts for most of the gap: 917 of the 1,189 reports marked REVIEWED name symbols, against 1 of the 3,489 marked UNREVIEWED, a status the guide describes as generally auto-generated, with no attempt made to determine packages or symbols.

The advisory and the macro share a shape: each is correct as written and applies only where a condition holds. Go's govulncheck tool tests that condition, using static analysis to narrow the reports down to those that could affect the program it is checking. When the condition is missing from the record, there is nothing to test and every importer looks exposed; the Go team's database documentation also notes that listing all versions as affected, when version mapping fails, trades missed warnings for false alarms on fixed versions. A macro library without recorded preconditions makes the same trade in a support dataset. Every case in the queue looks like a fit, so correct use and misuse arrive with identical labels.

The counts by review status come from a short script that reads every report file once, with no sampling:

What 4,710 vulnerability reports show about recording applicability
Review status in the report fileReportsName a symbol or derived symbolShare
REVIEWED1,18991777.1%
UNREVIEWED3,48910.03%
NEEDS_REVIEW322887.5%
All report files4,71094620.1%

The precondition card: write down what the template assumes

Grading fit starts with a card for every macro revision that lists, as checkable facts, the conditions under which sending it is correct. Each condition should be answerable yes, no or unknown from a named system, never from the tone of the transcript.

Preconditions come from the policy the macro implements and from the actions it performs. A refund-to-original-payment macro assumes the payment method still accepts credits; a replacement macro assumes the item shipped; a password reset assumes the account authenticates with a password. When the team cannot name a macro's preconditions, the card records the gap, and sends of that macro are graded undeterminable rather than quietly counted as correct.

Each card carries these fields:

  • Macro identifier and revision, with the dates that revision was live.
  • The intended situation in one sentence, written by the macro's owner.
  • Preconditions as yes-or-no facts, each tied to the system that can confirm it: billing, identity provider, order management or entitlement records.
  • Exclusions: situations where the macro must not go out even when every precondition holds, such as a fraud flag or a legal hold.
  • Actions the macro performs besides sending text: tags, status changes, assignment, snooze and field updates.
  • The expected end state and the window for observing it, such as refund settled, sign-in succeeded or replacement delivered.
  • A partial-fit rule saying what the agent must add when the case also needs something the macro does not do.

Annotated decision packet: a password-reset macro sent to a single sign-on account

This packet is illustrative: the vendor, account and timings are invented to show how grading works. A customer of a business software company writes that they cannot sign in. The agent applies the current, approved password-reset macro, which sends reset instructions, tags the conversation password_reset and closes it.

The template passes every check a wording review would make. The case fails the template's first precondition: the customer's workspace signs in through SAML single sign-on, so the vendor holds no password to reset and the customer's identity provider controls access. Two days later the same customer writes again, and a second agent points them to their own IT administrator, who restores access. The first conversation still reads as tagged, closed and thanked.

Each fact in the packet sits next to its source, so a grader or a model can reach the label without guessing:

Annotated decision packet: a password-reset macro sent to a single sign-on account
Packet fieldRecorded value (illustrative)Source systemGrader reading
Macro and revisionPassword reset, revision 7, live since MarchMacro library exportTemplate valid: current and approved
Precondition 1Account authenticates with a passwordPrecondition cardMust hold for the macro to apply
Case fact at send timeWorkspace uses SAML single sign-onIdentity settings snapshotPrecondition 1 fails
Actions appliedTag password_reset, status closedConversation event logThe macro's claim, not an outcome
Resulting stateNo successful sign-in for 48 hours; second contact on day 2Authentication log and contact historyExpected end state not reached
LabelMisuse: precondition failedGrader decisionDecided at send time; the second contact only confirms it

A grading rubric that scores fit apart from wording

Score six dimensions separately and derive the label from them, rather than asking a grader for one overall verdict. A single verdict blends the template's quality with the agent's choice, and that blend is what lets misuse pass as good work.

The label rules are mechanical. Correct use needs a valid template, every precondition confirmed and no exclusion triggered. Misuse is a valid template with a failed precondition, an ignored exclusion or an uncovered partial fit. A failed validity check sends the case to the stale-template label, which belongs to a drift dataset. When the record can neither confirm nor refute a precondition, the send is undeterminable and stays out of both training and scoring instead of defaulting to correct. The end state is graded on its own line, so a right choice that failed downstream is not mistaken for misuse.

The figure lays the two governing questions out as four cells, and the rubric below fills them in:

A grading rubric that scores fit apart from wording
DimensionQuestion for the graderEvidence that settles itA fail leads to
Template validityWas this revision current and approved for the policy governing this customer?Macro revision dates and the policy version in forceStale template, routed to the drift dataset
Precondition coverageDoes the card list the conditions the macro assumes?The macro's precondition cardUndeterminable until the card exists
ApplicabilityDid every precondition hold at send time?Snapshots from the systems the card namesMisuse: precondition failed
ExclusionsDid any do-not-send condition apply?Risk, legal and account flagsMisuse: exclusion ignored
CompletenessDid the case need something the macro does not do, and did the agent add it?Transcript checked against the partial-fit ruleMisuse: partial fit
End stateDid the expected state arrive within its window?Billing, identity, order or entitlement recordsRight choice, failed delivery: a separate label
A two-by-two grid crossing template validity with case fit: valid template and fitting case is correct use, valid template and non-fitting case is misuse, stale template is routed to drift, and a stale template on a non-fitting case is a double failure.
Template validity and case fit are graded separately; the misuse cell is the one a wording review passes.

Could a solved status or a thank-you settle fit instead?

Not for macro sends, because the macro often produced both signals. When the click that sends the text also closes the conversation, a closed status proves only that the macro ran. A thank-you usually arrives before anyone could know whether the refund settled or the sign-in worked; in the packet above, the customer thanked the agent for instructions that could never have worked.

The strongest form of the alternative deserves a test, not a dismissal. In a queue dominated by one macro whose preconditions almost always hold, closure with no repeat contact inside a fixed window may track fit closely enough to use. Check it on your own records: grade a sample of sends with the rubric, measure how often the cheap label agrees with the rubric label, and keep the cheap label only for macros where agreement is high and the disagreements do not cluster on one precondition. Report agreement per macro with its sample size, never as one blended figure.

Silence misleads in the other direction too. A customer who gave up, switched channels or got help from their own administrator may never write back, and an empty contact history then reads as success. That is why the end state is checked in the system where it happens, not inferred from what the customer did next.

Held-out tests a model cannot pass by memorizing macros

Split the evaluation by macro, not by ticket. When sends of the same macro sit in both training and test data, a model can learn which macros are usually right and score well without ever reading a precondition. Holding out whole macros, with their cards, forces it to compare an unseen card with an unseen case.

Keep the split deterministic so others can reproduce it: assign each macro to a fold from a hash of its identifier instead of a random draw, and publish the hash rule. Hold out a later period as well, since macro libraries and policies keep changing. Then report misuse rates per macro with their denominators, the share of sends graded undeterminable, and agreement between two independent graders on the applicability dimension.

This design cannot yet say how common misuse is in any real support operation, which macro types misfire most, or whether a model trained on precondition-graded sends picks macros better in production. Those answers need graded samples from real histories. The Go count measures how often a public security record names its applicability condition, not how often support macros fail; it is an analogy with numbers, not support evidence.

Turning the rubric into a data requirement

A buyer who wants to train or test macro selection should ask for histories that keep the macro library with its revisions, a send event naming the macro and revision used, and snapshots from the systems the precondition cards point to. Without that last part, a dataset shows which macros were used but cannot show whether they fit.

SourceX treats a request like this as a specification to source against: datasets for AI buyers come from U.S. businesses once a buyer asks for them, rather than from a shelf of ready-made sets, and the request form asks what the data must cover, how much of it, in which language and format, by when and under what license. A strong request names the macros in scope, the systems that hold precondition facts, the end states to verify and the acceptance test, such as rubric agreement on held-out macros. Whether any particular company's records can meet it is settled one case at a time.

Frequently asked questions

What is support macro misuse?

It is a send where the macro was current and accurate, but the case did not meet a condition the macro assumes, or needed something the macro does not do. It differs from a stale macro, whose promise no longer matches policy, and from a poor edit, where the agent's added wording is the problem. Labeling it requires the macro's preconditions and the case's facts at send time.

Can tags applied by a macro be used as training labels?

Only after their accuracy has been checked. A tag a macro applies records which macro the agent chose, so it is wrong exactly when the macro was misused. Grade a sample against precondition facts and keep the tags only for macros where they agree with the graded label.

Why split macro evaluation data by macro rather than by ticket?

If the same macro appears in both training and test data, a model can memorize which macros are usually right and never learn to check a precondition. Holding out whole macros, assigned to folds by a published hash of the macro identifier, makes the test measure fit judgment instead of recall.

Is a closed ticket with no repeat contact enough evidence that a macro fit?

Usually not, because the macro itself often closed the ticket and silent customers may simply have given up. It can serve as a cheap label for a specific macro if, on a graded sample, it agrees closely with the rubric label and its disagreements do not cluster on one precondition.

What does the Go vulnerability database have to do with support macros?

Both pair a correct general statement with a condition that decides whether it applies to one case. A Go report can name the vulnerable functions so tools flag only programs that use them; at commit e0bad56, 946 of 4,710 reports did. Recording preconditions for each macro does the same job for support data.

Sources

  • The Go vulnerability report format: when symbols are listed only programs using them are marked vulnerable, otherwise any importer of the module is; UNREVIEWED reports make no attempt to determine packages or symbols. Source
  • Report files counted for this article: 4,710 at commit e0bad56, 946 naming symbols or derived symbols, by review status as tabled. Source
  • govulncheck uses static analysis of source code or a binary's symbol table to narrow reports to those that could affect the application. Source
  • When version mapping fails, a Go report may conservatively list all versions as affected, avoiding false negatives but causing false positives on fixed versions. Source
  • Intercom macros can apply actions such as assign, tag, snooze, close and reopen, with or without a reply, and the close action runs after other actions. Source
  • Snorkel treats labeling functions as noisy heuristics with unknown accuracies and correlations, estimated without ground truth from their agreements and disagreements. Source
  • Outpatient progress notes written in 2018 at one academic medical center had a mean of 29.4% directly typed text, with 70.6% templated or copied (Rule et al., 2021). Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies