Skip to content

AI uses for records

Support unauthorized workaround dataset: when the fix broke policy

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Grade it on two separate axes: whether the customer's problem stayed resolved, and whether the action stayed within the policy version and authority in force at the time. A workaround that fixed the issue without approval is a resolved but noncompliant case, not a success. A useful dataset links the ticket, policy version, approval record, action log and later reopens or reversals so each label can be proven.

Agent run

Support unauthorized

This page covers support unauthorized workaround dataset: when the fix broke policy.Illustrative

Key takeaways

  • Label outcome and policy compliance separately; never let a good result excuse a missing approval.
  • A solved status is not an outcome; reopens, follow-ups and downstream reversals over a stated window are.
  • The authority record, an approval with role and timestamp, is what separates an exception from a workaround.
  • Store the policy version in force at action time, or the case cannot be labeled.
  • Use the compliant path as the target behavior and keep the historical workaround as a negative example with its outcome recorded.

How do you evaluate a support workaround that resolves an issue but violates policy?

Grade it on two separate axes: whether the customer's problem was actually and durably resolved, and whether the action stayed inside the policy version and authority in force at the time. A workaround that fixed the problem without the required check or approval is a resolved but noncompliant case. It is not a success, and a dataset that labels it as one teaches an agent that results excuse rule-breaking.

Public agent benchmarks show why both axes are needed. The original tau-bench retail policy tells the agent to authenticate the user before helping, even when the user offers an ID, and to list the details and get an explicit yes before any action that changes the database. Its grader compares the final database state with the expected state and checks required outputs. A state check alone can miss how the agent got there, which is exactly where an unauthorized workaround hides.

Real support histories contain these cases constantly. An agent waives a step to save a frustrated customer, a supervisor approves something in chat that never reaches the ticket, or a policy changes mid-week and half the team keeps using the old macro. Each of these can end with a happy customer, and each needs a different label.

Why a solved status and a grateful reply do not prove the outcome

A solved status records that an agent considered the ticket done; it does not record that the customer's problem went away. The outcome label has to come from evidence after the solve: whether the ticket was reopened, whether a follow-up arrived, and whether the downstream system shows the fix held.

Help desks keep some of this evidence by default. Zendesk's ticket metrics include a count of how many times a ticket was reopened and a count of public agent replies. Once a Zendesk ticket is closed it cannot be reopened, and a customer reply creates a follow-up ticket that references the closed one, so the outcome of a case can live on a different ticket ID.

The change history matters as much as the final state. Zendesk's ticket audits are a read-only history of every update to a ticket, each made of events such as field changes, comments and notifications. Salesforce adds an entry to the Case History related list when a tracked case field changes, with the date, time, change and user. Those histories are where you find the moment a status flipped, a field was overridden or a policy tag was removed.

Authorized exception, unauthorized workaround or policy gap?

Classify every off-policy action before labeling outcomes, because three different situations look alike in a transcript. The test is whether a record shows that someone with authority approved the deviation, and whether the policy covered the situation at all.

Absence of an approval is the hardest fact to prove, so define the search before labeling. Name the systems checked, such as the approval tool, the case history, internal notes and the supervisor queue, and the time span searched around the action. A case where that search was not possible is labeled unknown on the constraint axis, not unauthorized by default.

Stale-policy cases deserve their own category because they point at tooling rather than people. When a macro still carried last month's refund window, the agent followed the instructions in front of them; training a model to treat that as agent misconduct teaches the wrong lesson and hides a fixable process problem.

Authorized exception, unauthorized workaround or policy gap?
CategoryDefinitionEvidence that proves itTraining signal
Authorized exceptionPolicy allows deviation with approval, and approval was recorded before or at the actionApproval record with approver role and timestamp linked to the caseCorrect behavior when the approval path is followed
Unauthorized workaroundPolicy prohibits or requires approval, and no approval existsPolicy version in force, action log, absence of approval after a defined searchConstraint violation regardless of outcome
Retroactive approvalApproval was granted after the actionApproval timestamp later than the action timestampViolation with mitigation; never a positive example
Policy gapNo policy covered the situationPolicy text for the period shows no applicable ruleEscalation is the target behavior; route to policy owner
Stale policy appliedAgent followed a superseded versionPolicy version history and macro change logViolation caused by tooling; label separately from agent judgment

The outcome and constraint task pack, field by field

The task pack is the asset: one record per case that holds the starting state, the governing rule, the action taken, the authority behind it and the matured outcome, each pointing to the record that proves it. Fields that cannot be proven are marked unknown rather than filled with a plausible value.

At a glance

The outcome and constraint task pack, field by field

Version the pack and date every case. Policies change, and a case labeled against one version stays valid only for that version; when a policy changes, the pack keeps the old label and records which version it was judged against rather than relabeling history.

Build it from linked systems rather than the transcript. The conversation shows what the agent said; the action log, approval record and policy history show what was allowed and what was done.

The outcome and constraint task pack, field by field
FieldContentsTypical sourceValidation
case_id and linked_idsPrimary ticket, merged tickets, follow-ups, related orders or bugsHelp desk, order system, issue trackerEvery linked ID resolves; follow-ups traced
starting_stateCustomer request, account state and order state when the ticket openedHelp desk and system snapshots or field historyState reconstructed for the open time, not today
channelsEach channel used and handoffs between agents, bots and tiersTicket audits, routing eventsOrdered handoff list with timestamps
policy_versionPolicy or macro text in force at action time, with version IDKnowledge base history, macro change logEffective date precedes the action
promises_madeCommitments stated to the customerPublic repliesEach promise maps to a later action or a broken-promise flag
actions_takenBack-office actions with actor and timeAdmin logs, order and billing systemsMatches system state change
authority_recordApproval or override with approver role and time, or none foundApproval tool, case history, internal notesSearch scope recorded when none found
correctionsLater reversals, refunds of a mistaken action, data fixesField history, auditsEach correction linked to the action it undid
outcome_labelResolved durably, resolved temporarily, unresolved or unknownReopens, follow-ups, downstream recordsObservation window stated and elapsed
constraint_labelCompliant, authorized exception, unauthorized, retroactive or policy gapCategory rules aboveTwo reviewers agree or the case is adjudicated

Which label does each combination of outcome and compliance earn?

Keep outcome and compliance as two labels and derive the training target from the pair. Collapsing them into one score is how a dataset ends up rewarding the workaround.

Agreement on the constraint label should come before anyone looks at the outcome. Reviewers who already know the customer was happy tend to read ambiguous approvals generously, so the outcome evidence can be hidden from the first labeling pass and revealed only once the constraint label is set.

The derived target describes what an agent should have done, which is not always what the human agent did. For an unauthorized workaround that worked, the target behavior is usually the compliant path: request the approval, offer the permitted alternative or escalate. The historical action becomes a negative example, and its good outcome stays recorded so evaluators can see the tradeoff a strict policy imposed.

Which label does each combination of outcome and compliance earn?
OutcomeConstraintCase labelTarget agent behavior
Resolved durablyCompliant or authorized exceptionPositive exampleRepeat the action and its approval path
Resolved durablyUnauthorized or retroactiveConstraint failure with good outcomeSeek approval or escalate; do not copy the shortcut
Unresolved or temporaryCompliantPolicy limit caseFollow policy and escalate; flag the policy for review
Unresolved or temporaryUnauthorizedDouble failureNegative example on both axes
AnyPolicy gapEscalation caseHand off to a person with authority and say why
UnknownAnyExclude from scored setKeep for analysis only until evidence matures

Worked case (illustrative): a replacement shipped to an unverified address

No real customer, agent or order is behind this case; it is constructed to show the method. A customer reports that a delivered order arrived damaged and asks for a replacement sent to a different address because she is traveling. The policy version in force requires identity verification before any change of shipping address and supervisor approval for replacements shipped to a new address.

Starting state: order delivered 3 days earlier, account in good standing, one prior ticket. Linked evidence: the chat transcript, the order system's address change log, the replacement order, the macro used and the policy article version. The agent, under queue pressure, skipped verification, changed the address and shipped the replacement. The customer thanked the agent, and the ticket was solved within 20 minutes.

Decision against the rule: the address change happened with no verification step in the log and no approval record anywhere in the case history or approval tool after a search of both. Constraint label: unauthorized workaround. Outcome check after a 30-day window: no reopen, no follow-up ticket, and the replacement was delivered. Outcome label: resolved durably. Case label: constraint failure with good outcome. Target behavior: verify identity, then request supervisor approval or offer a replacement to the address on file.

Counterexample, kept in the same pack: a second case with the same request where the agent verified identity, a supervisor approved in the approval tool 4 minutes before the shipment, and the replacement shipped. That case is an authorized exception and a positive example. The two differ only in the authority record, which is the point: a model has to look for the approval, not infer it from the happy ending.

A third variant is worth adding when the records exist. The replacement shipped, but 9 days later the real account holder reported an order she never requested and a fraud review reversed it. That case is a double failure, and it shows why the observation window has to be long enough for this kind of downstream evidence to arrive.

How long after the solve can you call a workaround case resolved?

Durable resolution is proven by what did not happen during a stated observation window: no reopen, no follow-up ticket about the same problem, no reversal in the downstream system, and no new contact through another channel. The window is a per-category decision, short for a password reset and longer for a shipped replacement, and it must have fully elapsed before the label is set.

If the window has not elapsed when the pack is built, leave the outcome label as unknown and schedule a recheck. Filling it in early, on the strength of a thank-you message, is an easy way for a workaround to end up counted as a success.

Look past the help desk. A refund can be reversed in billing, a replacement can come back as a return, and a bug workaround can stop working after a release. Link the downstream record that would show the failure, and record that you checked it even when it shows nothing.

Follow-ups and merges need special care. Because a reply to a closed Zendesk ticket opens a follow-up rather than reopening the original, a reopen count of zero on the first ticket can hide a failure recorded on the second. Trace follow-up and merged tickets before setting the outcome label.

Where to find workaround cases without biasing the pack

Pull candidates from several signals at once, because each one finds a different slice of off-policy behavior and none finds all of it. Cases found only through quality assurance reviews over-represent the mistakes reviewers already look for; cases found only through complaints over-represent the workarounds that failed.

Stratify the pack by issue type, channel, tier and policy area, and keep compliant cases from the same strata alongside the deviations. Without those matched controls, a model can learn that a certain issue type always involves a workaround, which is a property of the sample and not of the work.

Have two reviewers who know the policy label each case independently, then compare. Disagreements usually point to an ambiguous policy clause or a missing approval record rather than a careless reviewer, and both are worth fixing before more cases are labeled. Record the adjudicated label and the reason, so a buyer can see how contested cases were settled.

  • Quality assurance scorecards that flagged a policy miss.
  • Override, manual adjustment or goodwill credit events in billing or order systems.
  • Tickets where a policy tag or macro was removed or changed after use, visible in ticket audits or case history.
  • Escalations and complaints that cite an agent's promise.
  • Fraud, chargeback or reversal records linked back to a support action.
  • Policy change announcements, with tickets from the weeks around each change.

Record gaps that make a workaround label impossible to prove

Some cases cannot be labeled honestly, and they should leave the scored set rather than receive a guessed label. The usual causes are records the support stack never kept.

  • The policy or macro text for the period was overwritten, so the version in force cannot be shown.
  • Approvals happened in chat or by phone and never reached the case, so authorized and unauthorized look identical.
  • Field history was not enabled for the relevant fields, or has aged out; Salesforce standard field history tracks a limited set of fields per object and keeps data for a limited period unless Field Audit Trail is used.
  • Tickets were merged and the merge dropped the original timestamps or channel.
  • Internal notes mix the approval record with customer personal data that cannot be separated cleanly.
  • The downstream system that would show a reversal is not linked to the ticket by any shared ID.

Can linked support cases and their policies be licensed?

Support teams that keep policy versions, approval records and linked order or billing histories hold the material these task packs need, and that linkage is rare in public data. SourceX runs the rights review, contracting, data preparation and delivery when a company licenses chosen support records as AI datasets.

Eligibility is checked first, through a company assessment that asks for no uploaded data. Customer personal data, agent identities and sensitive queues can be scoped out or de-identified before anything moves, because delivery waits until the company has approved the scope and terms and signed the agreement.

Frequently asked questions

What should an outcome and constraint task pack include?

Each case needs linked ticket and system IDs, the starting state, channels and handoffs, the policy version in force, promises made, actions taken, the authority record, corrections, and two separate labels for outcome and constraint. Every field points to the record that proves it, and unprovable fields are marked unknown.

Is a workaround that solved the problem ever a positive training example?

Only when a record shows it was an authorized exception, with an approval granted before or at the action by someone allowed to grant it. Without that, the case is a constraint failure even if the customer was happy, and the target behavior is the compliant path.

What proves a support problem stayed resolved after the ticket was closed?

The absence of reopens, follow-up tickets, downstream reversals and repeat contacts during an observation window that has fully elapsed. The window depends on the issue type, and follow-up and merged tickets must be traced before the outcome is labeled.

Which linked record proves the decision in a workaround case?

The authority record does: an approval or override with the approver's role and timestamp, tied to the case. The transcript shows what was said, but the approval tool, case history or ticket audits show whether the deviation was permitted.

How is this different from a general policy-following agent dataset?

A general dataset pairs conversations with policies and actions. This task pack focuses on the cases where outcome and compliance disagree, keeps them as two labels, and requires proof of authority and a matured outcome before a case is scored.

Sources

  • The original tau-bench retail policy requires the agent to authenticate the user and obtain explicit confirmation before consequential actions, and its grader compares the final database state with the expected state and checks required outputs. Source
  • In the current tau-bench repository each domain specifies a policy the agent must follow, the tools it can use and a set of evaluation tasks. Source
  • Zendesk's Ticket Metrics API reports the number of times a ticket was reopened and the number of public agent replies. Source
  • Zendesk ticket audits are a read-only history of all updates to a ticket, each update made of events such as field changes, comments and notifications. Source
  • Zendesk closed tickets cannot be reopened; a reply to a closed ticket creates a follow-up ticket that references the original. Source
  • Salesforce adds an entry to the Case History related list when a tracked case field changes, recording the date, time, nature of the change and the user. Source
  • Salesforce standard field history tracking allows up to 20 fields per object and retains history for up to 18 months unless Field Audit Trail is used. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies