Privacy and preparation
Complete customer case dataset: a necessity test for every joined record
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A complete customer case dataset holds the smallest packet from which an independent reviewer reaches the same decision and can check whether it held: the customer's request, the policy in force then, the action actually taken in other systems, and what happened afterward. Prove each element with a drop test, removing it to see whether reviewers' calls change, and leave out everything that fails.
Redaction
Complete customer case
Key takeaways
- The final ticket message records how a case ended, not what was allowed, what was done or whether it held.
- Complete means sufficient for an independent reviewer's decision and its verification, not every record that shares a customer ID.
- A drop test earns each element its place: remove it from a sample of packets and count how often blinded reviewers change their call.
- Each extra joined system adds distraction for models and linkage risk for people, so elements that fail the test stay out.
- Where history cannot be rebuilt, such as fields tracked only after a setting was switched on, declare the gap instead of filling it.
What makes a customer case dataset complete?
A customer case is complete when an independent reviewer, given only the packet, would reach the decision the agent faced and could check whether the outcome held. Completeness means sufficiency for that judgment, not volume: a packet that joins every record sharing a customer ID can be less useful in practice, because the decisive evidence gets buried.
Start from the decision, not the data. A complete customer case dataset for refund decisions needs different evidence from one for escalation routing or troubleshooting, so write down the decision the model must make, and the verdict a reviewer will give, before choosing a single record. The evidence that supports a useful dataset is then concrete: a written decision, a packet specification, drop-test results on a sample, and a declared list of gaps.
The method below has five parts: the questions every case must answer, a drop test that proves each element earns its place, an annotated illustrative packet for one refund dispute, exclusion rules, and how to record history that cannot be rebuilt.
What the final ticket message leaves out
The last message tells you how a case ended in words, not what was allowed, what was done or whether it worked. "Your refund has been processed" can follow a correct decision, a policy exception made under pressure, or a refund that never posted, and the message reads the same in all three cases.
Much of the decision evidence sits where the customer never looks. In Zendesk, every comment is either public or private, the private ones being internal notes, and each update to a ticket is stored as an audit representing that single change; even switching a public comment to private is logged as its own audit event. In Salesforce, the CaseHistory object holds historical information about changes made to a case. A dataset cut from customer-visible messages misses the internal reasoning and the change trail that show how the decision was reached.
The rest sits in other systems entirely: the refund posted in billing, the replacement shipped from the order system, the policy page as it read on the day. A model trained on closing messages learns to sound resolved, which is a different skill from resolving.
Five questions every case packet must answer
The questions split by time. What the customer asked and what was allowed must be fixed as of the moment the decision was made; what was done and whether it held draw on records created afterward, up to the end of an outcome window set per decision type before anyone looks at results. Who decided, and when, sits in between and usually comes from the change trail.
Every packet should let a reviewer answer five questions, and each answer should point either to specific evidence or to a declared gap. The map below links each question to the evidence that usually answers it, where that evidence tends to live, and what to record when it cannot be recovered.
| Question | Evidence that answers it | Typical source | If it cannot be recovered |
|---|---|---|---|
| What did the customer ask for? | The customer's own messages and attachments on this issue | Helpdesk thread, public comments | Exclude the case; without the request there is no task |
| What was allowed at the time? | Policy or entitlement text in force on the decision date, and the customer's plan then | Policy archive, subscription or contract record | Keep only with a gap label; never substitute today's policy |
| What was actually done? | The action record: refund, credit, replacement or account change | Billing, order or account system | Mark the outcome unverified |
| Did it hold? | Repeat contacts and reversals within a fixed window | Helpdesk search, billing adjustments | Mark the outcome immature or unknown |
| Who decided, and when? | Internal notes, assignee and status changes, approvals | Private comments, audits, field history | Keep with a gap label when the decision path matters to the task |
The drop test: proving each element belongs
An element belongs in the packet only if removing it changes what a careful reviewer concludes. To test that, draw a random sample of cases, build the full candidate packet for each, then give blinded reviewers versions with one element removed at a time and record whether their decision, or their verdict on the outcome, changes.
Write the rule down before running it. One workable form, with thresholds you choose and publish: keep an element if removing it changes the reviewer's call in at least an agreed share of sampled cases, or if it is the only evidence for one of the five questions; otherwise drop it. Run the test separately for each decision type, because an element that decides disputes may be noise for password resets. Measure reviewer agreement on full packets as well; if reviewers disagree with each other on complete packets, no element-level result means much.
Run it in reverse too. Starting from the core evidence, add one candidate element at a time and check whether calls improve. The two directions catch different errors: the drop test finds padding, and the add test finds context the core packet was missing.
Why not include everything and let the model sort it out? Because extra material is not neutral. Shi and colleagues (ICML 2023) found that adding irrelevant sentences to grade-school math problems sharply reduced model accuracy, and Liu and colleagues (TACL 2024) found that accuracy can drop significantly depending on where relevant information sits in a long input. Every unnecessary record joined into a case is another chance to bury the evidence that matters.
An annotated minimal packet for one refund dispute
The verdicts also decide where each element goes in the dataset. Decide elements are model input, because the agent could see them at the time. Verify elements are labels: repeat contacts and later status changes happened after the decision, so feeding them to a model as input leaks the outcome into the task. Keep the two in separate fields with timestamps, so an evaluator can check that no input postdates the decision.
In this example, the closing message alone, a polite confirmation of a partial refund, would teach a model nothing about the rule that set the amount or whether the customer accepted it.
The packet below is illustrative: an invented refund dispute on an annual subscription, showing which elements pass the drop test as decision evidence, which serve as verification, which are joined only on demand, and which stay out. Field names are generic; real systems differ.
| Element | Example content (illustrative) | Source | Verdict and reason |
|---|---|---|---|
| Customer request | Asks for a refund after an annual renewal charge, saying the reminder email never arrived | Helpdesk thread | Decide: without it there is no task |
| Policy in force then | The renewal refund rule as published on the charge date, with a 14-day window | Policy archive by date | Decide: removing it changes the call |
| Refund record | A partial refund posted three days after the agent's reply | Billing system | Decide: shows what was actually done |
| Status and owner history | Escalated to a billing specialist, then solved | Audits or field history | Verify: shows who decided and when |
| Repeat contacts | No new contact on this issue within 30 days | Helpdesk search by customer and issue | Verify: shows whether the outcome held |
| Earlier related case | A renewal complaint from the previous year | Helpdesk | On demand: joined only where the drop test shows it changes calls |
| Unrelated tickets, marketing email, full CRM profile | A shipping question, newsletters, lifetime account fields | Helpdesk, email platform, CRM | Left out: fails the drop test and widens linkage risk |
What stays out even when it is one join away
Leave out anything that fails the drop test, however easy the join. Other tickets from the same customer about unrelated issues, marketing and newsletter email, the full CRM profile, other customers named in a thread, and free-text fields with no bearing on the decision all add volume without adding evidence.
Two costs come on top of the noise. Where the GDPR applies to the personal data in a case, Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing, the data minimisation principle; a packet built by necessity is easier to defend under that standard than one built by convenience. And every joined system widens the linkage surface. NIST's report on de-identification, IR 8053, notes that researchers have shown some de-identified data can be re-identified, and describes a case where common movie ratings linked many records in a de-identified Netflix dataset with records in the Internet Movie Database, which had not been de-identified.
Whether a dataset needs GDPR analysis, and what de-identification is adequate, depends on the records and jurisdictions involved; counsel and a privacy reviewer decide that for the specific case. The drop test does not replace that review, but it hands the reviewer a short, justified list instead of a full export.
When history cannot be rebuilt, record the gap
If the evidence for one of the five questions no longer exists, say so in the packet instead of filling it in. Systems lose history in predictable ways. Salesforce field history tracking starts only from the date tracking is enabled for a field, so earlier changes are absent, and for long text fields over 255 characters it records that an edit happened without storing the before and after values.
Other common gaps include policy pages overwritten without an archive, refunds issued in a payment tool that never wrote back to the ticket, and attachments purged under a retention schedule. Label each with what is missing, why and since when, then apply the rule from the five-question map: exclude the case, keep it with a gap label, or keep it with an unverified or immature outcome.
Never substitute today's version of a policy or profile for the one that applied then. A packet that quietly fills gaps with current data teaches a model a decision rule that did not exist when the agent acted.
The strongest objection: long-context models can sort it out
The objection is that long-context models can separate relevant from irrelevant material on their own, so curation is wasted effort. Sometimes that is true, and the add test will show it: if extra elements improve reviewer or model calls, they belong in the packet.
The published evidence says capacity is not the same as use. Liu and colleagues found the position effect even in models built for long inputs, and in Shi and colleagues' tests distraction was large enough to need explicit mitigation, such as instructing the model to ignore irrelevant information. Add the privacy and linkage cost of joining more systems, and the burden of proof sits with each extra record, not with the minimal packet.
Turning the packet into a requirement a supplier can fill
A buyer who wants complete cases should send the decision, the packet specification and the acceptance test, not a request for all ticket data. Name the decision type, the five questions and the evidence for each, the outcome window, which elements are joined on demand, what must stay out, and how gaps are labeled. Then state how the delivery will be accepted: for example, drop-test results on an agreed sample, reviewer agreement on full packets, and a gap report by element.
At a glance
Turning the packet into a requirement a supplier can fill
SourceX works from requests of this kind rather than from a catalog: it looks for U.S. businesses that can meet a buyer's stated need, and its request form asks what scope and volume you need, in which language and format, by when, and under what licensing terms. The company supplying the records approves the scope and terms and signs before any data reaches SourceX, and de-identification is carried out by SourceX directly or through its processing partner. None of this implies that a particular case set exists or will pass a given acceptance test; that is settled request by request.
A requirement that covers the following gives a supplier something specific to scope against:
Checklist · 7 items
- The decision type, and the reviewer verdict the model's output will be judged against
- The five questions, with the evidence and source system expected for each
- The decision timestamp rule and the outcome window for each decision type
- Elements joined only on demand, and the drop-test result that justifies each one
- Exclusions: systems, record types and fields that must never be joined
- Gap labels, and the rule for excluding, flagging or downgrading an incomplete case
- The acceptance test: drop-test sample size, reviewer agreement target and the format of the gap report
0 of 7 ticked. Tick each item as you go.
Frequently asked questions
What is a complete customer case in a support dataset?
It is the smallest set of evidence from which an independent reviewer could reach the decision the agent faced and check whether the outcome held. That usually means the customer's request, the policy in force at the time, the action taken in other systems and repeat contacts within a fixed window, with any gaps declared.
Why is the final ticket message not enough for AI training?
Because it records how the case ended, not what was allowed or done. The same closing message can follow a correct decision, a policy exception or a refund that never posted. Internal notes, change history and records in billing or order systems carry the decision evidence.
Should a case dataset include internal notes?
Usually, when the task depends on how the decision was made. Helpdesks such as Zendesk store internal notes as private comments that customers never see, so they hold reasoning missing from the public thread. They also tend to carry more personal and employee information, so they need the same drop test and privacy review as any other element.
How far back should case history go?
Only as far as the drop test justifies. Earlier cases from the same customer belong when removing them changes reviewers' calls, as in a repeat complaint; otherwise they add noise and linkage risk. Set the window per decision type and write it into the specification.
What if part of a case's history was never recorded?
Label the gap rather than reconstructing it from current data. Salesforce, for instance, tracks field history only from the date tracking was switched on for a field. Decide in advance whether a case missing a given element is excluded, kept with a gap label, or kept with an unverified outcome.
Sources
- Zendesk ticket comments are either public or private, with private comments serving as internal notes. Source
- Zendesk stores an audit for every ticket update, each audit representing a single change. Source
- Zendesk records a public comment being made private as its own audit event. Source
- Salesforce's CaseHistory object holds historical information about changes made to a case. Source
- Salesforce field history tracking starts only when enabled and stores no before and after values for long text fields over 255 characters. Source
- Shi and colleagues (ICML 2023) found irrelevant context sharply reduces language model accuracy on math problems. Source
- Liu and colleagues (TACL 2024) found model performance degrades with the position of relevant information in long inputs. Source
- GDPR Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing. Source
- NIST IR 8053 notes that some de-identified data can be re-identified, including by linking records to other datasets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.