Fine-tuning LLMs for classification and routing: labeled outcome data

This guide sits in the fine-tuning and post-training data hub and focuses on one need: replacing a long routing or intent prompt with a small fine-tuned model trained on labeled outcomes. For licensing the underlying record types, see the owner pages for IT service management ticket datasets and customer support ticket datasets.

Which system fields make usable classification targets

The best classification targets are fields a human or downstream process acted on, not fields filled in at intake. In an ITSM tool such as ServiceNow or Jira Service Management, the assignment group at resolution, the resolution code and the closed category are stronger than the caller-selected category on the portal form. In a CRM or support desk such as Salesforce Service Cloud or Zendesk, the final disposition, the escalation flag and the macro applied are better than the initial "reason for contact" dropdown.

Ask suppliers which of these fields exist and how each is populated:

• Routing labels: final assignment group, queue, tier, owning team.
• Intent and category labels: category and subcategory at close, product area, issue type.
• Outcome labels: resolution code, disposition, refund approved or denied, claim paid or denied, escalated or not.
• Priority labels: final priority or severity after triage, which often differs from the requester's choice.

Public intent benchmarks show the shape you are aiming for. BANKING77, for example, pairs customer service queries with 77 fine-grained intents in a single domain [3]. Operational data differs in one important way: its labels were produced by work, not by annotators following a guideline, so their reliability has to be verified rather than assumed. Our page on verifying outcome labels in operational records covers that test in more depth.

Why system-of-record labels are noisy

System labels are noisy because they serve operations, not model training, and every operational shortcut leaves a trace in the label column. Treat each of the following as a hypothesis to test on the supplier's data, not a certainty.

• Reassignment ping-pong. A ticket bounces between three groups before landing in the right one. If the export holds only the first assignment, your target is the wrong team for a large share of hard cases.
• Catch-all classes. "General inquiry", "Other" and "Miscellaneous" absorb records agents did not bother to classify. These classes are often large, heterogeneous and impossible to learn.
• Taxonomy drift. Categories are renamed, merged or split during a tool migration or reorganization. A 2022 "Billing > Disputes" and a 2025 "Payments > Chargebacks" may be the same class.
• Default-value bias. Required dropdowns with a preselected value inflate whichever option sits first.
• Automation labels. Auto-close rules, bots and assignment rules write labels without human judgment, so a model learns the rule rather than the decision.
• Late relabeling. Some teams fix categories at month-end for reporting, so the closed label may not match what the record looked like at triage time.

Label error is not unique to operational data: widely used benchmark test sets also contain measurable label errors, enough to change model rankings [2]. The practical difference is that you can often recover a better label from operational history if the supplier exports it.

Reconstructing the final label from reassignment history

Use the final label after reassignment as the training target, and keep the reassignment history as metadata. The final assignment group is usually the closest proxy for "who could actually resolve this", while the history tells you which classes are confusable and which records were genuinely ambiguous.

Request the audit or history table (in ServiceNow, the sys_audit or metric records for assignment changes; in Jira, the changelog) alongside the record table. Then derive features such as hop count, time in each group and whether the category changed after assignment.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "record_id": "INC-000123",
  "created_at": "2024-03-04T09:12:00Z",
  "text": "VPN client drops every 10 minutes since the laptop refresh. Error 809.",
  "channel": "portal",
  "initial_category": "Hardware",
  "initial_assignment_group": "Service Desk L1",
  "assignment_history": [
    {"group": "Service Desk L1", "at": "2024-03-04T09:12:00Z"},
    {"group": "Endpoint Engineering", "at": "2024-03-04T11:40:00Z"},
    {"group": "Network Access", "at": "2024-03-05T08:05:00Z"}
  ],
  "final_assignment_group": "Network Access",
  "final_category": "Network > Remote access",
  "resolution_code": "Config change",
  "closed_by_automation": false,
  "taxonomy_version": "2024-01",
  "pii_treatment": "names and asset tags replaced with placeholders"
}

From records like this, a training example uses the text and the final group. Records with three or more hops, automation closes or a category edited after close can be routed to a review queue instead of the training set.

Mapping label taxonomies across suppliers and over time

If you combine records from several suppliers, or from one supplier across a taxonomy change, you need an explicit crosswalk from source labels to your target taxonomy. Without it, the same intent ends up under different labels and the model learns noise between them.

Illustrative example: invented to show structure; it does not describe an available dataset.

One-to-many rows are where most effort goes. Either relabel a sample by hand and train a splitter, or keep the coarser class and accept lower granularity. Record the taxonomy version on every row so you can retrain when the crosswalk changes.

Class balance and rare classes in routing data

Routing data is almost always long-tailed: a handful of queues hold most of the volume, and the classes that matter most, such as fraud, safety or legal escalations, are rare. Decide your sampling strategy before you set volume requirements with a supplier.

• Ask for the label distribution per class and per month before agreeing on scope, not after delivery.
• Specify a minimum count per class you care about and allow oversampling of rare classes from a longer time window.
• Keep a natural-distribution test set so you can report precision and recall at real prevalence; a balanced test set overstates rare-class precision.
• Track per-class metrics and the confusion matrix, not only accuracy, because a model that ignores a 1% class can still score 99%.

How many examples per class you need depends on the base model size and class difficulty; our guide to how much data you need to fine-tune an LLM covers the planning logic.

Measuring label noise before you buy

You can estimate label noise on a sample before committing to the full purchase, and you should. A disciplined noise audit turns a vague quality claim into numbers you can put in a scope document.

Illustrative example: invented to show structure; it does not describe an available dataset.

Label-noise audit checklist

1. Draw a stratified sample, for example 50 to 100 records per class, including rare classes and catch-alls.
2. Have two domain reviewers relabel the text blind to the system label, using your target taxonomy; compute agreement between reviewers and between each reviewer and the system label.
3. Train a quick baseline (a small encoder or a few-shot LLM) and run confident learning to flag records whose given label is unlikely given the model's out-of-sample predictions [1].
4. Compare the flag rate across months and teams to locate drift or individual-agent habits.
5. Check for near-duplicates and templated text, such as auto-generated alerts or macros, which inflate apparent accuracy and leak across splits [4].
6. Measure the share of records closed by automation, labeled with a default value, or in a catch-all class.
7. Write the results into acceptance criteria: per-class relabel agreement, maximum catch-all share, required history fields.

The checklist for evaluating a fine-tuning dataset before buying adds general quality tests that apply alongside this one.

Formatting labeled records for a small-model fine-tune

Format each record as an instruction-style example whose output is exactly one label string from a closed set. Managed tuning services such as Vertex AI expect labeled examples in a documented JSONL structure [5]; open-source stacks accept similar chat or prompt-completion formats.

• Put the allowed labels in the system prompt or keep them fixed in training so the model learns a constrained output; for multi-label or hierarchical outputs, see structured-output fine-tuning data.
• Split train, validation and test by time, not randomly, so the test set reflects future traffic and catches taxonomy drift.
• Deduplicate before splitting; templated tickets in both train and test overstate performance [4].
• Strip signatures, disclaimers and quoted email threads that carry routing cues the model should not rely on, such as a team's email footer.

A useful baseline is the prompt you are replacing. If a fine-tuned small model does not beat the prompted large model on your time-split test set, more data will not fix a label problem.

Rights and privacy questions specific to routing data

Ticket and case text routinely contains customer names, emails, phone numbers, account numbers and free-text health or financial details, so privacy treatment is part of label quality. Ask how identifiers were removed and whether replacement tokens preserve class-relevant cues; replacing every product SKU with a generic token can erase the signal that separates two queues.

Provenance matters as much for business records as for public datasets. An audit of more than 1,800 public text datasets found licenses frequently missing or wrong on hosting sites [6], so ask for documented ownership, consents and allowed uses for every source rather than relying on a label in a data card. Our AI training data compliance hub collects the relevant regulations and records.

How SourceX sources labeled operational records

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, and finance and legal workflows, and manages the licensing and ongoing purchases. Nothing is held in stock: you describe the records and label fields you need, SourceX looks for US businesses that hold them, and a request does not guarantee a match. The process runs Find, Assess, Agree, Transact and Manage, and nothing is contracted until a supplier agrees.

Every dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Personal details such as names, emails, phones and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. You can describe your routing or intent taxonomy and label requirements on the SourceX buyers page.

Request labeled records for classifier fine-tuning

If you are replacing a routing prompt with a fine-tuned classifier, describe the record type, label fields and history you need. SourceX looks for US businesses that hold matching operational data, and every release is approved by the supplying company. Describe your classification data need to SourceX.