AI uses for records
Support unauthorized workaround dataset: when the fix broke policy
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Grade it on two separate axes: whether the customer's problem stayed resolved, and whether the action stayed within the policy version and authority in force at the time. A workaround that fixed the issue without approval is a resolved but noncompliant case, not a success. A useful dataset links the ticket, policy version, approval record, action log and later reopens or reversals so each label can be proven.
Agent run
Support unauthorized
Key takeaways
- Label outcome and policy compliance separately; never let a good result excuse a missing approval.
- A solved status is not an outcome; reopens, follow-ups and downstream reversals over a stated window are.
- The authority record, an approval with role and timestamp, is what separates an exception from a workaround.
- Store the policy version in force at action time, or the case cannot be labeled.
- Use the compliant path as the target behavior and keep the historical workaround as a negative example with its outcome recorded.
How do you evaluate a support workaround that resolves an issue but violates policy?
Grade it on two separate axes: whether the customer's problem was actually and durably resolved, and whether the action stayed inside the policy version and authority in force at the time. A workaround that fixed the problem without the required check or approval is a resolved but noncompliant case. It is not a success, and a dataset that labels it as one teaches an agent that results excuse rule-breaking.
Public agent benchmarks show why both axes are needed. The original tau-bench retail policy tells the agent to authenticate the user before helping, even when the user offers an ID, and to list the details and get an explicit yes before any action that changes the database. Its grader compares the final database state with the expected state and checks required outputs. A state check alone can miss how the agent got there, which is exactly where an unauthorized workaround hides.
Real support histories contain these cases constantly. An agent waives a step to save a frustrated customer, a supervisor approves something in chat that never reaches the ticket, or a policy changes mid-week and half the team keeps using the old macro. Each of these can end with a happy customer, and each needs a different label.
Why a solved status and a grateful reply do not prove the outcome
A solved status records that an agent considered the ticket done; it does not record that the customer's problem went away. The outcome label has to come from evidence after the solve: whether the ticket was reopened, whether a follow-up arrived, and whether the downstream system shows the fix held.
Help desks keep some of this evidence by default. Zendesk's ticket metrics include a count of how many times a ticket was reopened and a count of public agent replies. Once a Zendesk ticket is closed it cannot be reopened, and a customer reply creates a follow-up ticket that references the closed one, so the outcome of a case can live on a different ticket ID.
The change history matters as much as the final state. Zendesk's ticket audits are a read-only history of every update to a ticket, each made of events such as field changes, comments and notifications. Salesforce adds an entry to the Case History related list when a tracked case field changes, with the date, time, change and user. Those histories are where you find the moment a status flipped, a field was overridden or a policy tag was removed.
Authorized exception, unauthorized workaround or policy gap?
Classify every off-policy action before labeling outcomes, because three different situations look alike in a transcript. The test is whether a record shows that someone with authority approved the deviation, and whether the policy covered the situation at all.
Absence of an approval is the hardest fact to prove, so define the search before labeling. Name the systems checked, such as the approval tool, the case history, internal notes and the supervisor queue, and the time span searched around the action. A case where that search was not possible is labeled unknown on the constraint axis, not unauthorized by default.
Stale-policy cases deserve their own category because they point at tooling rather than people. When a macro still carried last month's refund window, the agent followed the instructions in front of them; training a model to treat that as agent misconduct teaches the wrong lesson and hides a fixable process problem.
| Category | Definition | Evidence that proves it | Training signal |
|---|---|---|---|
| Authorized exception | Policy allows deviation with approval, and approval was recorded before or at the action | Approval record with approver role and timestamp linked to the case | Correct behavior when the approval path is followed |
| Unauthorized workaround | Policy prohibits or requires approval, and no approval exists | Policy version in force, action log, absence of approval after a defined search | Constraint violation regardless of outcome |
| Retroactive approval | Approval was granted after the action | Approval timestamp later than the action timestamp | Violation with mitigation; never a positive example |
| Policy gap | No policy covered the situation | Policy text for the period shows no applicable rule | Escalation is the target behavior; route to policy owner |
| Stale policy applied | Agent followed a superseded version | Policy version history and macro change log | Violation caused by tooling; label separately from agent judgment |
The outcome and constraint task pack, field by field
The task pack is the asset: one record per case that holds the starting state, the governing rule, the action taken, the authority behind it and the matured outcome, each pointing to the record that proves it. Fields that cannot be proven are marked unknown rather than filled with a plausible value.
At a glance
The outcome and constraint task pack, field by field
Version the pack and date every case. Policies change, and a case labeled against one version stays valid only for that version; when a policy changes, the pack keeps the old label and records which version it was judged against rather than relabeling history.
Build it from linked systems rather than the transcript. The conversation shows what the agent said; the action log, approval record and policy history show what was allowed and what was done.
| Field | Contents | Typical source | Validation |
|---|---|---|---|
| case_id and linked_ids | Primary ticket, merged tickets, follow-ups, related orders or bugs | Help desk, order system, issue tracker | Every linked ID resolves; follow-ups traced |
| starting_state | Customer request, account state and order state when the ticket opened | Help desk and system snapshots or field history | State reconstructed for the open time, not today |
| channels | Each channel used and handoffs between agents, bots and tiers | Ticket audits, routing events | Ordered handoff list with timestamps |
| policy_version | Policy or macro text in force at action time, with version ID | Knowledge base history, macro change log | Effective date precedes the action |
| promises_made | Commitments stated to the customer | Public replies | Each promise maps to a later action or a broken-promise flag |
| actions_taken | Back-office actions with actor and time | Admin logs, order and billing systems | Matches system state change |
| authority_record | Approval or override with approver role and time, or none found | Approval tool, case history, internal notes | Search scope recorded when none found |
| corrections | Later reversals, refunds of a mistaken action, data fixes | Field history, audits | Each correction linked to the action it undid |
| outcome_label | Resolved durably, resolved temporarily, unresolved or unknown | Reopens, follow-ups, downstream records | Observation window stated and elapsed |
| constraint_label | Compliant, authorized exception, unauthorized, retroactive or policy gap | Category rules above | Two reviewers agree or the case is adjudicated |
Which label does each combination of outcome and compliance earn?
Keep outcome and compliance as two labels and derive the training target from the pair. Collapsing them into one score is how a dataset ends up rewarding the workaround.
Agreement on the constraint label should come before anyone looks at the outcome. Reviewers who already know the customer was happy tend to read ambiguous approvals generously, so the outcome evidence can be hidden from the first labeling pass and revealed only once the constraint label is set.
The derived target describes what an agent should have done, which is not always what the human agent did. For an unauthorized workaround that worked, the target behavior is usually the compliant path: request the approval, offer the permitted alternative or escalate. The historical action becomes a negative example, and its good outcome stays recorded so evaluators can see the tradeoff a strict policy imposed.
| Outcome | Constraint | Case label | Target agent behavior |
|---|---|---|---|
| Resolved durably | Compliant or authorized exception | Positive example | Repeat the action and its approval path |
| Resolved durably | Unauthorized or retroactive | Constraint failure with good outcome | Seek approval or escalate; do not copy the shortcut |
| Unresolved or temporary | Compliant | Policy limit case | Follow policy and escalate; flag the policy for review |
| Unresolved or temporary | Unauthorized | Double failure | Negative example on both axes |
| Any | Policy gap | Escalation case | Hand off to a person with authority and say why |
| Unknown | Any | Exclude from scored set | Keep for analysis only until evidence matures |
Worked case (illustrative): a replacement shipped to an unverified address
No real customer, agent or order is behind this case; it is constructed to show the method. A customer reports that a delivered order arrived damaged and asks for a replacement sent to a different address because she is traveling. The policy version in force requires identity verification before any change of shipping address and supervisor approval for replacements shipped to a new address.
Starting state: order delivered 3 days earlier, account in good standing, one prior ticket. Linked evidence: the chat transcript, the order system's address change log, the replacement order, the macro used and the policy article version. The agent, under queue pressure, skipped verification, changed the address and shipped the replacement. The customer thanked the agent, and the ticket was solved within 20 minutes.
Decision against the rule: the address change happened with no verification step in the log and no approval record anywhere in the case history or approval tool after a search of both. Constraint label: unauthorized workaround. Outcome check after a 30-day window: no reopen, no follow-up ticket, and the replacement was delivered. Outcome label: resolved durably. Case label: constraint failure with good outcome. Target behavior: verify identity, then request supervisor approval or offer a replacement to the address on file.
Counterexample, kept in the same pack: a second case with the same request where the agent verified identity, a supervisor approved in the approval tool 4 minutes before the shipment, and the replacement shipped. That case is an authorized exception and a positive example. The two differ only in the authority record, which is the point: a model has to look for the approval, not infer it from the happy ending.
A third variant is worth adding when the records exist. The replacement shipped, but 9 days later the real account holder reported an order she never requested and a fraud review reversed it. That case is a double failure, and it shows why the observation window has to be long enough for this kind of downstream evidence to arrive.
How long after the solve can you call a workaround case resolved?
Durable resolution is proven by what did not happen during a stated observation window: no reopen, no follow-up ticket about the same problem, no reversal in the downstream system, and no new contact through another channel. The window is a per-category decision, short for a password reset and longer for a shipped replacement, and it must have fully elapsed before the label is set.
If the window has not elapsed when the pack is built, leave the outcome label as unknown and schedule a recheck. Filling it in early, on the strength of a thank-you message, is an easy way for a workaround to end up counted as a success.
Look past the help desk. A refund can be reversed in billing, a replacement can come back as a return, and a bug workaround can stop working after a release. Link the downstream record that would show the failure, and record that you checked it even when it shows nothing.
Follow-ups and merges need special care. Because a reply to a closed Zendesk ticket opens a follow-up rather than reopening the original, a reopen count of zero on the first ticket can hide a failure recorded on the second. Trace follow-up and merged tickets before setting the outcome label.
Where to find workaround cases without biasing the pack
Pull candidates from several signals at once, because each one finds a different slice of off-policy behavior and none finds all of it. Cases found only through quality assurance reviews over-represent the mistakes reviewers already look for; cases found only through complaints over-represent the workarounds that failed.
Stratify the pack by issue type, channel, tier and policy area, and keep compliant cases from the same strata alongside the deviations. Without those matched controls, a model can learn that a certain issue type always involves a workaround, which is a property of the sample and not of the work.
Have two reviewers who know the policy label each case independently, then compare. Disagreements usually point to an ambiguous policy clause or a missing approval record rather than a careless reviewer, and both are worth fixing before more cases are labeled. Record the adjudicated label and the reason, so a buyer can see how contested cases were settled.
- Quality assurance scorecards that flagged a policy miss.
- Override, manual adjustment or goodwill credit events in billing or order systems.
- Tickets where a policy tag or macro was removed or changed after use, visible in ticket audits or case history.
- Escalations and complaints that cite an agent's promise.
- Fraud, chargeback or reversal records linked back to a support action.
- Policy change announcements, with tickets from the weeks around each change.
Record gaps that make a workaround label impossible to prove
Some cases cannot be labeled honestly, and they should leave the scored set rather than receive a guessed label. The usual causes are records the support stack never kept.
- The policy or macro text for the period was overwritten, so the version in force cannot be shown.
- Approvals happened in chat or by phone and never reached the case, so authorized and unauthorized look identical.
- Field history was not enabled for the relevant fields, or has aged out; Salesforce standard field history tracks a limited set of fields per object and keeps data for a limited period unless Field Audit Trail is used.
- Tickets were merged and the merge dropped the original timestamps or channel.
- Internal notes mix the approval record with customer personal data that cannot be separated cleanly.
- The downstream system that would show a reversal is not linked to the ticket by any shared ID.
Can linked support cases and their policies be licensed?
Support teams that keep policy versions, approval records and linked order or billing histories hold the material these task packs need, and that linkage is rare in public data. SourceX runs the rights review, contracting, data preparation and delivery when a company licenses chosen support records as AI datasets.
Eligibility is checked first, through a company assessment that asks for no uploaded data. Customer personal data, agent identities and sensitive queues can be scoped out or de-identified before anything moves, because delivery waits until the company has approved the scope and terms and signed the agreement.
Frequently asked questions
What should an outcome and constraint task pack include?
Each case needs linked ticket and system IDs, the starting state, channels and handoffs, the policy version in force, promises made, actions taken, the authority record, corrections, and two separate labels for outcome and constraint. Every field points to the record that proves it, and unprovable fields are marked unknown.
Is a workaround that solved the problem ever a positive training example?
Only when a record shows it was an authorized exception, with an approval granted before or at the action by someone allowed to grant it. Without that, the case is a constraint failure even if the customer was happy, and the target behavior is the compliant path.
What proves a support problem stayed resolved after the ticket was closed?
The absence of reopens, follow-up tickets, downstream reversals and repeat contacts during an observation window that has fully elapsed. The window depends on the issue type, and follow-up and merged tickets must be traced before the outcome is labeled.
Which linked record proves the decision in a workaround case?
The authority record does: an approval or override with the approver's role and timestamp, tied to the case. The transcript shows what was said, but the approval tool, case history or ticket audits show whether the deviation was permitted.
How is this different from a general policy-following agent dataset?
A general dataset pairs conversations with policies and actions. This task pack focuses on the cases where outcome and compliance disagree, keeps them as two labels, and requires proof of authority and a matured outcome before a case is scored.
Sources
- The original tau-bench retail policy requires the agent to authenticate the user and obtain explicit confirmation before consequential actions, and its grader compares the final database state with the expected state and checks required outputs. Source
- In the current tau-bench repository each domain specifies a policy the agent must follow, the tools it can use and a set of evaluation tasks. Source
- Zendesk's Ticket Metrics API reports the number of times a ticket was reopened and the number of public agent replies. Source
- Zendesk ticket audits are a read-only history of all updates to a ticket, each update made of events such as field changes, comments and notifications. Source
- Zendesk closed tickets cannot be reopened; a reply to a closed ticket creates a follow-up ticket that references the original. Source
- Salesforce adds an entry to the Case History related list when a tracked case field changes, recording the date, time, nature of the change and the user. Source
- Salesforce standard field history tracking allows up to 20 fields per object and retains history for up to 18 months unless Field Audit Trail is used. Source
See if your company qualifies
A short company assessment. No data uploads are needed.