Software companies
AI support agents: why your historical resolutions decide how well they work
By SourceX Editorial · Updated
Short answer
AI support agents resolve cases well only when they can draw on historical resolutions that record what actually fixed each problem: solved tickets with the final answer, current macros, a maintained knowledge base and escalation notes. The working rule: check whether your solved tickets capture the real fix before you trust any vendor's resolution rate.
Key takeaways
- An AI support agent can only repeat fixes that your records actually captured.
- Closed is not the same as resolved: auto-closed, merged and spam tickets should be filtered out before they ground or test an agent.
- Escalation notes and linked engineering issues hold the fixes that never reached the knowledge base.
- Test any agent against a held-out set of your own past tickets, not only against a vendor's reported resolution rate.
- The linked resolution histories that improve your own agent are the same records AI developers license for training and evaluation.
What does an AI support agent draw on when it answers?#
An AI support agent draws on whatever your support operation has written down: knowledge base articles, macros, past ticket conversations and the notes agents left when a case was escalated. Most deployments ground the agent on these sources at answer time rather than training a model from scratch, so the quality of the records sets the ceiling on the quality of the answers.
Each source teaches something different. The knowledge base describes how the product is meant to work. Solved tickets show how problems were actually fixed for real customers, including workarounds nobody documented. Escalation records show where frontline answers ran out and what engineering or a senior agent did next.
| Input | What it gives the agent | Common gap |
|---|---|---|
| Solved tickets with the final reply | Real fixes, phrased for customers, across edge cases | Fix recorded only in an internal note or a side chat |
| Macros and saved replies | Approved wording and standard steps | Macros written for old product versions and never retired |
| Knowledge base articles | Intended product behavior and setup steps | Articles that contradict what agents actually tell customers |
| Escalation and handoff notes | Where frontline answers fail and why | Handoffs with no summary of what was already tried |
| Linked engineering issues | Root cause and the release that fixed it | Ticket closed before the fix shipped, link never added |
| Outcome fields: CSAT, reopen flag, resolution code | Whether the answer worked | Fields left blank or set in bulk at closing |
Why do historical resolutions matter more than the knowledge base?#
Historical resolutions matter more because they record what worked, while the knowledge base records what was supposed to work. In most support teams the two drift apart: agents find a workaround for a billing sync error, share it in a team channel and use it for months before anyone updates the article.
An agent grounded only on the knowledge base inherits that drift. It gives the documented answer, the customer replies that it did not help, and the case lands with a human anyway. An agent that can also see how similar tickets were closed has a chance of offering the workaround first.
Resolutions also carry judgment that articles leave out: when to offer a credit, when to ask for logs, when a symptom points to an outage rather than a configuration mistake. Those calls are what support leaders most want an agent to get right, and they mostly exist only in ticket threads.
How can you tell whether your ticket history records the real fix?#
Ticket history records the real fix when a reader who was not on the case can see the problem, what was tried and what finally worked, without opening another system. A quick sample answers the question better than any dashboard: pull a few dozen solved tickets from your largest categories and read each one end to end.
Note how often the final public reply contains the actual steps, and how often it says only that the issue was fixed on our side. Then check the fields and links against the list below.
- The final public reply states the steps or the change that resolved the issue.
- A resolution code or category is chosen by the agent, not left at a default.
- Internal notes explain the root cause when the public reply does not.
- Tickets caused by bugs link to the Jira or Linear issue and, ideally, the release.
- Reopened tickets keep their full thread rather than starting a new one.
- Tickets closed for inactivity, merged as duplicates or marked as spam are tagged so they can be filtered out.
What does a vendor's resolution rate actually measure?#
A vendor's resolution rate often counts conversations that ended without a handoff to a human, which is not the same as problems that were solved. Definitions vary between vendors, so the same agent can post very different numbers depending on what counts as resolved. A customer who gives up, opens a new ticket the next day or picks up the phone can still count as resolved under some definitions.
Ask each vendor for its written definition, then run your own test. Hold back a set of past solved tickets from your highest-volume categories that the agent has not seen, replay each customer's first message, and have experienced agents grade every answer as correct, partly correct, wrong or unsafe against the fix that actually worked. That test only works if your history records the fix.
| What a dashboard may count | What to check in your own records |
|---|---|
| Conversation ended without a handoff | Did the same customer reopen or file a related ticket soon after? |
| Customer clicked a positive rating | Does the reply match how agents resolved similar tickets? |
| Article suggested and viewed | Was the article current for the customer's product version? |
| Ticket auto-closed after no reply | Was the issue fixed, or did the customer simply leave? |
Illustrative: a scheduling software company audits its resolutions first#
Illustrative: a fictional workforce scheduling software company plans to put an AI agent in front of its Zendesk queue. Before choosing a vendor, the head of support samples solved tickets from billing, account access, integrations and mobile app categories and checks each one for a documented fix.
Billing and account access tickets are clean: macros are current and final replies spell out the steps. Integration tickets are not. Many were closed with a note saying engineering resolved it, while the real explanation sits in Jira comments and a Slack escalation channel. Mobile tickets refer to app versions that were retired.
The team launches the agent on billing and account access only, archives macros for retired app versions, and makes a root-cause note and Jira link required on integration escalations. It keeps a held-out set of past integration tickets as a test bank, so the agent can be measured against real fixes before that category is switched on.
Why the same records interest AI developers#
The linked resolution histories that make your own agent work are the same records AI developers look for when they train and evaluate support agents in general. A ticket that connects a customer's question to an escalation, an engineering fix and a confirmed outcome shows a complete piece of service work, which generic chat data does not.
Licensing those records is a separate decision from deploying your own agent. The company keeps ownership, licenses a prepared copy for a defined use and decides what is in scope. Customer contracts, privacy notices and the helpdesk vendor's terms need review first, and personal and confidential details are removed before anything leaves the company.
Automated de-identification helps but is not enough on its own. Presidio, an open-source PII detection tool, warns in its documentation that automated detection cannot guarantee finding all sensitive information and that additional protections should be used. That is why preparation pairs tools with human review of ticket threads, where customers paste account numbers, addresses and screenshots.
How SourceX approaches support resolution records#
SourceX treats support resolution histories as one of the stronger record families a software company holds. In the SourceX Enterprise Data Value Framework, tickets that link to escalations, engineering fixes and confirmed outcomes add human-generated signal and AI utility, consistently filled outcome fields count toward data cleanliness, and years of exportable history count toward scale and recency. Customer personal details add privacy burden and preparation cost.
Deals run through the SourceX five-step transaction: Supply, Rights, Preparation, Approval and Delivery. The fit check needs only metadata, such as the helpdesk name, ticket categories and date coverage, and the supplier approves each step before prepared records leave the company.
Frequently asked questions
Should we clean up old tickets before switching on an AI agent?
Clean the inputs the agent will use, not the history itself. Retire outdated macros, fix or archive stale articles, and tag tickets that were auto-closed, merged or spam so they can be filtered out. Editing old ticket threads destroys the record of what really happened, which you need both for testing your own agent and for any later licensing review.
Is recent history more useful than older tickets?
Recent tickets match the current product, so they ground answers best. Older tickets still help when they are tagged with the product version or plan they refer to, because they show how recurring problems were handled over time. Without version tags, old resolutions can teach an agent fixes for screens and settings that no longer exist.
Should the agent be able to read internal notes?
Internal notes often hold the real fix, but they also hold candid comments, customer details and security information. Many teams let the agent use summarized resolutions rather than raw notes, and keep notes out of customer-facing answers. Decide this with your security and privacy leads before connecting the helpdesk.
Does licensing ticket history affect our own AI agent rollout?
Not by itself. A license gives an AI developer access to a prepared copy of selected records for a defined use, and your helpdesk, your history and your own agent stay as they are. Check that any exclusivity or field-of-use terms leave internal use open. The two efforts share groundwork, since both reward tickets that record the problem, the fix and the outcome.
Who should own resolution quality in a support team?
Usually support operations, working with whoever owns the knowledge base. They set required fields at closing, review samples on a regular cadence and decide when a workaround becomes an article. When nobody owns it, resolution codes drift and the agent inherits the drift.
Sources
- Presidio's documentation warns that because it uses automated detection mechanisms, there is no guarantee it will find all sensitive information, so additional systems and protections should be employed. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.