Skip to content

Privacy and preparation

Complete customer case dataset: a necessity test for every joined record

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A complete customer case dataset holds the smallest packet from which an independent reviewer reaches the same decision and can check whether it held: the customer's request, the policy in force then, the action actually taken in other systems, and what happened afterward. Prove each element with a drop test, removing it to see whether reviewers' calls change, and leave out everything that fails.

Redaction

Complete customer case

This page covers complete customer case dataset: a necessity test for every joined record.Illustrative

Key takeaways

  • The final ticket message records how a case ended, not what was allowed, what was done or whether it held.
  • Complete means sufficient for an independent reviewer's decision and its verification, not every record that shares a customer ID.
  • A drop test earns each element its place: remove it from a sample of packets and count how often blinded reviewers change their call.
  • Each extra joined system adds distraction for models and linkage risk for people, so elements that fail the test stay out.
  • Where history cannot be rebuilt, such as fields tracked only after a setting was switched on, declare the gap instead of filling it.

What makes a customer case dataset complete?

A customer case is complete when an independent reviewer, given only the packet, would reach the decision the agent faced and could check whether the outcome held. Completeness means sufficiency for that judgment, not volume: a packet that joins every record sharing a customer ID can be less useful in practice, because the decisive evidence gets buried.

Start from the decision, not the data. A complete customer case dataset for refund decisions needs different evidence from one for escalation routing or troubleshooting, so write down the decision the model must make, and the verdict a reviewer will give, before choosing a single record. The evidence that supports a useful dataset is then concrete: a written decision, a packet specification, drop-test results on a sample, and a declared list of gaps.

The method below has five parts: the questions every case must answer, a drop test that proves each element earns its place, an annotated illustrative packet for one refund dispute, exclusion rules, and how to record history that cannot be rebuilt.

What the final ticket message leaves out

The last message tells you how a case ended in words, not what was allowed, what was done or whether it worked. "Your refund has been processed" can follow a correct decision, a policy exception made under pressure, or a refund that never posted, and the message reads the same in all three cases.

Much of the decision evidence sits where the customer never looks. In Zendesk, every comment is either public or private, the private ones being internal notes, and each update to a ticket is stored as an audit representing that single change; even switching a public comment to private is logged as its own audit event. In Salesforce, the CaseHistory object holds historical information about changes made to a case. A dataset cut from customer-visible messages misses the internal reasoning and the change trail that show how the decision was reached.

The rest sits in other systems entirely: the refund posted in billing, the replacement shipped from the order system, the policy page as it read on the day. A model trained on closing messages learns to sound resolved, which is a different skill from resolving.

Five questions every case packet must answer

The questions split by time. What the customer asked and what was allowed must be fixed as of the moment the decision was made; what was done and whether it held draw on records created afterward, up to the end of an outcome window set per decision type before anyone looks at results. Who decided, and when, sits in between and usually comes from the change trail.

Every packet should let a reviewer answer five questions, and each answer should point either to specific evidence or to a declared gap. The map below links each question to the evidence that usually answers it, where that evidence tends to live, and what to record when it cannot be recovered.

Five questions every case packet must answer
QuestionEvidence that answers itTypical sourceIf it cannot be recovered
What did the customer ask for?The customer's own messages and attachments on this issueHelpdesk thread, public commentsExclude the case; without the request there is no task
What was allowed at the time?Policy or entitlement text in force on the decision date, and the customer's plan thenPolicy archive, subscription or contract recordKeep only with a gap label; never substitute today's policy
What was actually done?The action record: refund, credit, replacement or account changeBilling, order or account systemMark the outcome unverified
Did it hold?Repeat contacts and reversals within a fixed windowHelpdesk search, billing adjustmentsMark the outcome immature or unknown
Who decided, and when?Internal notes, assignee and status changes, approvalsPrivate comments, audits, field historyKeep with a gap label when the decision path matters to the task

The drop test: proving each element belongs

An element belongs in the packet only if removing it changes what a careful reviewer concludes. To test that, draw a random sample of cases, build the full candidate packet for each, then give blinded reviewers versions with one element removed at a time and record whether their decision, or their verdict on the outcome, changes.

Write the rule down before running it. One workable form, with thresholds you choose and publish: keep an element if removing it changes the reviewer's call in at least an agreed share of sampled cases, or if it is the only evidence for one of the five questions; otherwise drop it. Run the test separately for each decision type, because an element that decides disputes may be noise for password resets. Measure reviewer agreement on full packets as well; if reviewers disagree with each other on complete packets, no element-level result means much.

Run it in reverse too. Starting from the core evidence, add one candidate element at a time and check whether calls improve. The two directions catch different errors: the drop test finds padding, and the add test finds context the core packet was missing.

Why not include everything and let the model sort it out? Because extra material is not neutral. Shi and colleagues (ICML 2023) found that adding irrelevant sentences to grade-school math problems sharply reduced model accuracy, and Liu and colleagues (TACL 2024) found that accuracy can drop significantly depending on where relevant information sits in a long input. Every unnecessary record joined into a case is another chance to bury the evidence that matters.

An annotated minimal packet for one refund dispute

The verdicts also decide where each element goes in the dataset. Decide elements are model input, because the agent could see them at the time. Verify elements are labels: repeat contacts and later status changes happened after the decision, so feeding them to a model as input leaks the outcome into the task. Keep the two in separate fields with timestamps, so an evaluator can check that no input postdates the decision.

In this example, the closing message alone, a polite confirmation of a partial refund, would teach a model nothing about the rule that set the amount or whether the customer accepted it.

The packet below is illustrative: an invented refund dispute on an annual subscription, showing which elements pass the drop test as decision evidence, which serve as verification, which are joined only on demand, and which stay out. Field names are generic; real systems differ.

An annotated minimal packet for one refund dispute
ElementExample content (illustrative)SourceVerdict and reason
Customer requestAsks for a refund after an annual renewal charge, saying the reminder email never arrivedHelpdesk threadDecide: without it there is no task
Policy in force thenThe renewal refund rule as published on the charge date, with a 14-day windowPolicy archive by dateDecide: removing it changes the call
Refund recordA partial refund posted three days after the agent's replyBilling systemDecide: shows what was actually done
Status and owner historyEscalated to a billing specialist, then solvedAudits or field historyVerify: shows who decided and when
Repeat contactsNo new contact on this issue within 30 daysHelpdesk search by customer and issueVerify: shows whether the outcome held
Earlier related caseA renewal complaint from the previous yearHelpdeskOn demand: joined only where the drop test shows it changes calls
Unrelated tickets, marketing email, full CRM profileA shipping question, newsletters, lifetime account fieldsHelpdesk, email platform, CRMLeft out: fails the drop test and widens linkage risk
Three concentric rings for decide, verify and on-demand evidence in a customer case packet, with the elements of the illustrative refund dispute assigned to each ring and three elements listed as left out.
The illustrative refund dispute packet: what each ring carries and what stays out.

What stays out even when it is one join away

Leave out anything that fails the drop test, however easy the join. Other tickets from the same customer about unrelated issues, marketing and newsletter email, the full CRM profile, other customers named in a thread, and free-text fields with no bearing on the decision all add volume without adding evidence.

Two costs come on top of the noise. Where the GDPR applies to the personal data in a case, Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing, the data minimisation principle; a packet built by necessity is easier to defend under that standard than one built by convenience. And every joined system widens the linkage surface. NIST's report on de-identification, IR 8053, notes that researchers have shown some de-identified data can be re-identified, and describes a case where common movie ratings linked many records in a de-identified Netflix dataset with records in the Internet Movie Database, which had not been de-identified.

Whether a dataset needs GDPR analysis, and what de-identification is adequate, depends on the records and jurisdictions involved; counsel and a privacy reviewer decide that for the specific case. The drop test does not replace that review, but it hands the reviewer a short, justified list instead of a full export.

When history cannot be rebuilt, record the gap

If the evidence for one of the five questions no longer exists, say so in the packet instead of filling it in. Systems lose history in predictable ways. Salesforce field history tracking starts only from the date tracking is enabled for a field, so earlier changes are absent, and for long text fields over 255 characters it records that an edit happened without storing the before and after values.

Other common gaps include policy pages overwritten without an archive, refunds issued in a payment tool that never wrote back to the ticket, and attachments purged under a retention schedule. Label each with what is missing, why and since when, then apply the rule from the five-question map: exclude the case, keep it with a gap label, or keep it with an unverified or immature outcome.

Never substitute today's version of a policy or profile for the one that applied then. A packet that quietly fills gaps with current data teaches a model a decision rule that did not exist when the agent acted.

The strongest objection: long-context models can sort it out

The objection is that long-context models can separate relevant from irrelevant material on their own, so curation is wasted effort. Sometimes that is true, and the add test will show it: if extra elements improve reviewer or model calls, they belong in the packet.

The published evidence says capacity is not the same as use. Liu and colleagues found the position effect even in models built for long inputs, and in Shi and colleagues' tests distraction was large enough to need explicit mitigation, such as instructing the model to ignore irrelevant information. Add the privacy and linkage cost of joining more systems, and the burden of proof sits with each extra record, not with the minimal packet.

Turning the packet into a requirement a supplier can fill

A buyer who wants complete cases should send the decision, the packet specification and the acceptance test, not a request for all ticket data. Name the decision type, the five questions and the evidence for each, the outcome window, which elements are joined on demand, what must stay out, and how gaps are labeled. Then state how the delivery will be accepted: for example, drop-test results on an agreed sample, reviewer agreement on full packets, and a gap report by element.

At a glance

Turning the packet into a requirement a supplier can fill

SourceX works from requests of this kind rather than from a catalog: it looks for U.S. businesses that can meet a buyer's stated need, and its request form asks what scope and volume you need, in which language and format, by when, and under what licensing terms. The company supplying the records approves the scope and terms and signs before any data reaches SourceX, and de-identification is carried out by SourceX directly or through its processing partner. None of this implies that a particular case set exists or will pass a given acceptance test; that is settled request by request.

A requirement that covers the following gives a supplier something specific to scope against:

Checklist · 7 items

  1. The decision type, and the reviewer verdict the model's output will be judged against
  2. The five questions, with the evidence and source system expected for each
  3. The decision timestamp rule and the outcome window for each decision type
  4. Elements joined only on demand, and the drop-test result that justifies each one
  5. Exclusions: systems, record types and fields that must never be joined
  6. Gap labels, and the rule for excluding, flagging or downgrading an incomplete case
  7. The acceptance test: drop-test sample size, reviewer agreement target and the format of the gap report

0 of 7 ticked. Tick each item as you go.

Frequently asked questions

What is a complete customer case in a support dataset?

It is the smallest set of evidence from which an independent reviewer could reach the decision the agent faced and check whether the outcome held. That usually means the customer's request, the policy in force at the time, the action taken in other systems and repeat contacts within a fixed window, with any gaps declared.

Why is the final ticket message not enough for AI training?

Because it records how the case ended, not what was allowed or done. The same closing message can follow a correct decision, a policy exception or a refund that never posted. Internal notes, change history and records in billing or order systems carry the decision evidence.

Should a case dataset include internal notes?

Usually, when the task depends on how the decision was made. Helpdesks such as Zendesk store internal notes as private comments that customers never see, so they hold reasoning missing from the public thread. They also tend to carry more personal and employee information, so they need the same drop test and privacy review as any other element.

How far back should case history go?

Only as far as the drop test justifies. Earlier cases from the same customer belong when removing them changes reviewers' calls, as in a repeat complaint; otherwise they add noise and linkage risk. Set the window per decision type and write it into the specification.

What if part of a case's history was never recorded?

Label the gap rather than reconstructing it from current data. Salesforce, for instance, tracks field history only from the date tracking was switched on for a field. Decide in advance whether a case missing a given element is excluded, kept with a gap label, or kept with an unverified outcome.

Sources

  • Zendesk ticket comments are either public or private, with private comments serving as internal notes. Source
  • Zendesk stores an audit for every ticket update, each audit representing a single change. Source
  • Zendesk records a public comment being made private as its own audit event. Source
  • Salesforce's CaseHistory object holds historical information about changes made to a case. Source
  • Salesforce field history tracking starts only when enabled and stores no before and after values for long text fields over 255 characters. Source
  • Shi and colleagues (ICML 2023) found irrelevant context sharply reduces language model accuracy on math problems. Source
  • Liu and colleagues (TACL 2024) found model performance degrades with the position of relevant information in long inputs. Source
  • GDPR Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing. Source
  • NIST IR 8053 notes that some de-identified data can be re-identified, including by linking records to other datasets. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies