AI uses for records
Support product version reasoning data: how to test version-specific answers
By SourceX Editorial · Updated
10 sections
Short answer
A support evaluation should test product-version-specific answers by pinning every task to the customer's exact version, deployment and the support policy in force on the case date, then grading the reply against an answer key written for that version. Pair tasks that ask the same question on different versions, add tasks where the version is unknown, and score resolution only after the outcome has matured.
Agent run
Support product version
Key takeaways
- A reply that is right for the current release but wrong for the customer's release is a wrong answer and should be graded as one.
- Each task needs the version at case creation, the evidence for it, the policy revision in force and an answer key written for that combination.
- Contrast pairs, fix-boundary, unreleased-fix, policy-change, end-of-life and unknown-version tasks expose version handling that typical cases hide.
- Where agents act, grade the end state of records as well as the reply, and repeat runs to measure consistency.
- Label resolution only after a maturity window, linking reopenings, follow-up tickets and released fix versions.
How should a support evaluation test product-version-specific answers?
Treat the product version as part of the question, not as background. Each evaluation task should carry the customer's version, edition and deployment, the date the case was opened, and the policy text that applied on that date, and the grader should hold an answer key written for exactly that combination.
A reply that is correct for the current release but wrong for the customer's release is a wrong answer. So is a reply that promises a fix from a version the customer cannot install, or quotes a refund window that changed after the case was opened. An evaluation that grades against a single, current answer rewards exactly these mistakes.
Three design choices make the test honest. Build contrast pairs, where the same customer question has different correct answers on two versions. Include tasks where the version is missing, so the correct move is to ask or check before answering. And label whether the problem actually stayed solved only after enough time has passed to see reopenings and follow-up tickets.
Why the same support question has several correct answers
Correct support answers are situated: they depend on when and where the question is asked. Research on open-domain question answering found the same pattern at scale. The SituatedQA authors estimate that roughly 16.5 percent of questions in the NQ-Open set have answers that depend on time or place, and that models trained on older data lost roughly 15 points of accuracy on questions asked in the present, even when given an updated evidence corpus.
Software makes the dependency explicit. Under Semantic Versioning, a MAJOR increment signals incompatible API changes, a MINOR increment adds backward-compatible functionality and a PATCH increment carries backward-compatible bug fixes. A question such as whether an endpoint accepts a parameter can have a yes answer on one minor line and a no on the next major line.
Vendors also run several release lines at once, each with different promises. The Node.js release schedule, for example, places each line in a Current, Active LTS or Maintenance phase, and the Maintenance phase is limited to critical bug fixes and security updates. Whether a fix will arrive on a customer's line is therefore a version question, not a general one.
Where the version and policy facts already sit in support records
Most of the facts a version-pinned task needs already exist, spread across the help desk, the issue tracker and the release notes. The work is to find them for each case and record which system each fact came from.
The table lists the usual locations and the trap each one sets. Check these against your own configuration, because field names and settings differ by account.
Reconstruct each fact as of the case date. If the help desk keeps field change history or an audit log, read the version value from the first event on the ticket rather than the current row. If it does not, look for the version in the customer's first message, an attached log or the license record, and record that the value was reported rather than confirmed. A task built from today's field value can quietly turn a 4.x case into a 5.x case and grade a correct historical reply as wrong.
| Fact the task needs | Typical record | What to read | Trap to check |
|---|---|---|---|
| Customer's product version | Help desk custom field, such as a Zendesk drop-down ticket field | Value on the ticket at creation, not today | Agents often update the field after an upgrade, overwriting the version the case was about |
| Versions where a defect occurs | Jira Affects versions on the linked bug | All versions listed, not just the latest | The field supports search by version but does not add the issue to a release |
| Version that ships the fix | Jira Fix versions on the linked bug | Fix version plus its release status and date | An unreleased fix version is a plan, not an answer a customer can act on |
| What changed between versions | Changelog or release notes | Added, Changed, Deprecated, Removed, Fixed and Security entries | Yanked releases are sometimes left out of changelogs entirely |
| Support status of the line | Published release or lifecycle schedule | Phase and end-of-life date on the case date | A line in maintenance may receive security fixes only |
| Policy in force | Versioned policy page, macro or knowledge article | The revision effective on the case date | Macros are edited in place, so the old wording may survive only in sent replies |
What should the version and effective policy task dataset brief include?
The brief is the contract between the people who assemble tasks and the people who grade them. Every task row should let a grader decide, without guessing, what the right answer was for that customer on that day and which records prove it.
At a glance
What should the version and effective policy task dataset brief include
Use the fields below as is, or rename them to match your warehouse. A task that cannot fill the required fields belongs in the unknown-version or excluded set, not in the graded set.
Write the answer key as required points and forbidden claims rather than a model reply. Required points are the facts a correct answer must contain, such as the version where the behavior changed and the supported workaround. Forbidden claims are the tempting wrong statements, such as a fix date for an unreleased version or a policy term that was retired. Short keys of this kind let two graders agree, and they survive rewording that would break a string match.
Version the brief itself. When the product team corrects a release note or a policy owner clarifies a transition rule, update the affected keys, bump the brief version and keep the old one, so past scores remain reproducible.
| Field | What it holds | Format or allowed values | Required |
|---|---|---|---|
| task_id and case_ids | Stable task key plus every ticket, chat and call merged into the case | Task key; list of source record IDs across channels | Yes |
| case_opened_at | When the customer first raised the problem | ISO 8601 timestamp with time zone | Yes |
| product_version_at_case | Version, edition and deployment the customer ran when the case opened | Exact version string; cloud, self-hosted or mobile | Yes, or mark unknown |
| version_evidence | Where the version came from and how sure it is | Ticket field, log line, license record or customer statement; confirmed or reported | Yes |
| policy_version_id | The support, refund or SLA policy revision in force on the case date | Policy name plus revision and effective dates | When the answer depends on policy |
| release_facts | Changelog entries, affected versions and fix versions that bear on the question | Links to release notes and issue keys with release status | When the answer depends on behavior |
| answer_key | The correct resolution for this version and policy, written by a product expert | Short required points plus forbidden claims | Yes |
| contrast_task_id | The sibling task that asks the same question on another version | Task key or none | For contrast pairs |
| handoffs_and_corrections | Escalations, engineering replies and any agent correction of an earlier wrong answer | Ordered list with timestamps and roles | When present |
| outcome_label and outcome_matured_at | Whether the problem stayed solved and when that was judged | Resolved, reopened, workaround, unresolved or not yet mature; date | Yes |
Building contrast pairs and the other task types
A graded set built only from typical cases will mostly test whether the model knows the newest release. Deliberately mix task types so that version handling is what separates good answers from weak ones.
Keep the mix visible in the brief and report scores by task type. A model can score well overall while failing every unknown-version task, which is the failure most likely to cause harm in production.
Source every task from a real case where you can. Contrast pairs work best when both halves come from genuine tickets, because customers on older releases describe problems differently, attach different logs and often do not know their own version. Where one half must be written by hand, label it as constructed so analysts can check whether constructed tasks score differently from real ones.
| Task type | What it tests | How to build it | Correct behavior |
|---|---|---|---|
| Contrast pair | Whether the answer changes when only the version changes | Two real cases with the same question on versions whose behavior differs per the changelog | Different, version-correct answers to each |
| Fix boundary | Whether the agent knows which release carries a fix | Case on a version before the fix version, with a later released fix | Name the fixed version and the upgrade path, or a workaround |
| Unreleased fix | Whether the agent promises what does not exist yet | Case where the fix version is still unreleased on the case date | No dated promise; state the known workaround and status |
| Policy change | Whether the agent applies the policy in force on the case date | Case opened just before or after a policy revision | Quote the revision that applied to that case |
| End-of-life line | Whether the agent recognizes a line outside support | Case on a line past end-of-life or in maintenance only | Explain the support status and the supported path |
| Unknown version | Whether the agent checks before answering | Case where no record establishes the version | Ask for or look up the version first |
Grading a reply against the version-pinned answer key
Grade the substance first: did the reply identify or confirm the version, state behavior that matches the release facts for that version, apply the policy revision in force and avoid every forbidden claim in the key. Tone and length come after these checks, never instead of them.
At a glance
Grading a reply against the version-pinned answer key
Where the agent can act, grade the end state as well as the words. The tau-bench benchmark gives agents a written domain policy and tools, then judges success by comparing the database state at the end of the conversation with an annotated goal state. Its published retail policy, for instance, requires explicit customer confirmation before consequential actions such as cancellations or returns. A support task can be judged the same way: the correct ticket fields, the correct linked issue and no action the policy forbids.
Run each task more than once. The tau-bench authors introduced pass^k, the chance that all of k independent attempts succeed, because agents that solve a task once often fail it on a repeat. Version handling is a common source of that inconsistency, so report it per task type.
- Hard fail: the reply asserts behavior or a fix that the release facts contradict for the customer's version.
- Hard fail: the reply applies a policy revision that was not in force on the case date.
- Hard fail: the reply answers an unknown-version task without asking or checking.
- Partial: the substance is right but the reply omits the upgrade path or workaround the key requires.
- Pass: every required point present, no forbidden claim, and any action leaves the correct end state.
Proving a fixed problem stayed fixed after the ticket closed
A closed status proves only that someone closed the ticket. Resolution is shown by what happened afterward: the case was not reopened, the customer did not open a new ticket on the same problem, and any fix the reply relied on actually shipped in a released version the customer could run.
Set a maturity window before labeling, and record the date the label was judged. The window length is a choice for your product; a team might use, as an illustration, 30 days for a configuration question and longer when the answer depended on an upcoming release. Cases still inside the window are labeled not yet mature and kept out of the graded outcome set.
Link the evidence, not just the verdict. A resolved label should point to the absence of reopenings in the window, the linked issue's released fix version where one was promised, and, if available, the version the customer later reported. Corrections matter too: when an agent gave a wrong first answer and a colleague fixed it, keep both turns, because the correction is the most useful supervision in the case.
Worked example: one export question across three releases
This example is illustrative; the product, versions and counts are invented to show the method. A software company runs 4.2 self-hosted, 4.3 cloud and 5.0 cloud. Customers repeatedly ask whether audit logs can be exported to CSV.
The release facts give three different answers. The 4.3 changelog lists CSV audit export under Added. The 5.0 changelog lists the old export endpoint under Removed and a new one under Added. The 4.2 line is in maintenance, receiving security fixes only, and has no export feature.
The team builds a contrast set of three tasks from real cases, one per version, each with its own answer key. A fourth task comes from a case where the customer never stated a version and the ticket field was blank; its key requires the agent to check the account or ask before answering.
The exception is a case where an agent told a 4.2 customer the feature was coming in a patch. The linked bug's fix version was 4.3, not a 4.2 patch, and the customer reopened the case 12 days later. That case enters the set with both the wrong first answer and the corrected reply, and its outcome is labeled reopened, not resolved.
When the version cannot be pinned: narrow, relabel or drop the task
Some cases will not support a version-pinned key, and forcing one in corrupts the set. Decide case by case using the evidence you have.
At a glance
When the version cannot be pinned: narrow, relabel or drop the task
Bring in a product expert when release notes and the issue tracker disagree about which version carries a fix, and a privacy reviewer before any customer-written text leaves the support system.
- Narrow the task when the version is known but the policy revision is not: grade only the behavior points and drop the policy points from the key.
- Relabel the task as unknown-version when no record establishes the version; its key then grades whether the agent asked or checked.
- Drop the task when the release facts conflict and no expert can settle which answer was correct on the case date.
- Hold the outcome label when the case is inside the maturity window or the promised fix version is still unreleased.
- Exclude cases whose text includes credentials, payment card data or other content your privacy review rules out.
Licensing version-pinned support cases as AI data
Support histories that keep the version field, the linked engineering issues, the release notes and the corrections are rarer than raw ticket dumps, and the version links are what make them useful for evaluation. Before anyone outside the company sees them, separate four questions: who can access the systems, what the privacy notice permits, who owns customer-written content and what a license would actually grant. De-identification does not by itself create a right to license.
Support organizations can license selected cases like these through SourceX, which looks after rights review, contracting, preparation, delivery and payment. Buyers typically want steady history, often two or more years and tens of thousands of records in one system, though rarer material can still qualify, with every dataset weighed individually. A brief assessment, with no records uploaded, shows whether a company qualifies.
Frequently asked questions
Should the version field come from the ticket or from product telemetry?
Prefer the most reliable record available on the case date and note which one you used. A ticket field set by the customer is a report, while a license record or log line is usually a confirmation. When the two disagree, keep both in the version evidence and let a product expert decide which applies before the task is graded.
How many contrast pairs does a version-aware support evaluation need?
There is no fixed number; aim for enough pairs per task type that a score change is not one or two cases moving. Report results by task type so a small pair set is visible rather than hidden in an overall score. Add pairs first for the questions customers ask most often across versions.
Can synthetic tasks replace real cases for version testing?
Synthetic tasks help fill gaps, such as an end-of-life line with few tickets, but they inherit the author's assumptions about how customers describe versions. Real cases carry the messy evidence, missing fields and mid-case upgrades that make version handling hard. Label synthetic tasks clearly and score them separately.
What if the policy changed while the case was still open?
Record both revisions and the date of the change in the task. The answer key should state which revision governs, which is usually set by the policy's own transition rule or by the date the customer first raised the issue. If the rule is unclear, have the policy owner decide before the task is graded.
Does an end-of-life version make the case useless for evaluation?
No. Cases on unsupported lines test whether an agent recognizes the support status and explains the supported path instead of troubleshooting as usual. Keep them as their own task type with keys that reward the correct explanation.
Sources
- SituatedQA estimates roughly 16.5% of NQ-Open questions have context-dependent answers and finds models trained on past data drop roughly 15 points in accuracy on present-day questions even with an updated evidence corpus. Source
- Semantic Versioning 2.0.0 increments MAJOR for incompatible API changes, MINOR for backward-compatible functionality and PATCH for backward-compatible bug fixes. Source
- The Node.js release schedule places release lines in Current, Active LTS or Maintenance phases, with Maintenance limited to critical bug fixes and security updates. Source
- Keep a Changelog 1.1.0 groups changes as Added, Changed, Deprecated, Removed, Fixed and Security and notes that yanked releases often do not appear in changelogs. Source
- Jira's Fix versions field records the release that ships the work, while Affects versions records the versions a bug affects and enables search without adding the issue to that version. Source
- Zendesk administrators create custom ticket fields such as drop-down lists, each option has a unique tag, and a field's type cannot be changed after creation. Source
- The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with an annotated goal state and introduces the pass^k reliability metric. Source
- The tau-bench retail agent policy requires explicit user confirmation before consequential actions that update the database, such as cancel, modify, return or exchange. Source
See if your company qualifies
A short company assessment. No data uploads are needed.