Skip to content

AI uses for records

New customer cold start coverage: is your evaluation dataset too familiar?

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Measure how much history the agent could see for each case, at the moment the case happened, and stratify the evaluation by that depth. Score first-contact, thin-history and long-history customers separately, keep each customer in only one split, and reweight the strata to the mix you expect in deployment. A business archive usually overrepresents repeat customers, so new customer cold start coverage is the gap to check first.

Evaluation split

Dataset new customer cold

This page covers new customer cold start coverage: is your evaluation dataset too familiar.Illustrative

Key takeaways

  • Repeat customers give an agent context that first-time customers do not, so archives built from them flatter agent scores.
  • Compute history depth as of each case's timestamp and as the production agent would see it, including identity-match limits.
  • Sample equal counts per depth stratum, then reweight to the deployment mix and report both totals.
  • Keep each customer in one split and test on a later period so familiarity cannot leak into the score.
  • Look for cases the archive never captured, such as abandoned first contacts and churned customers, before trusting any stratum.

How do you account for repeat-customer familiarity in an agent benchmark?

Treat customer history depth as a variable you measure and control, not background noise. For every case, compute how many prior interactions the agent could have seen at the time of the case, assign it to a depth stratum, and report the agent's score per stratum. Then weight the strata to the population the agent will actually serve and state that weighting next to the result.

Two further controls keep familiarity from inflating the number. Keep each customer entirely in training or entirely in evaluation, so the agent is not tested on people it has effectively memorized. And check what the archive is missing: first contacts that never became a record, customers who left, and outcomes too recent to judge.

This guide gives a customer history depth stratified evaluation design you can apply to support, sales, service or order records: how to define the population, how to compute depth correctly, a design table, splitting rules, a worked example with illustrative numbers, and the cases where stratifying by depth misleads.

Why repeat customers make easier tasks than new customers

A repeat customer's case arrives with context. The agent can read prior tickets, past orders, known preferences, earlier promises and the resolution that worked last time. A first-time customer's case has none of that: the agent must ask, infer and verify, and it is more likely to face an onboarding question, an identity check or a product it has not seen before.

Business archives lean toward the easy side. A long-standing customer generates many records over the years while a first contact generates one, so a random sample of cases overweights familiar customers even if most of the business's growth comes from new ones. An agent tuned and tested on that sample can post a strong overall score that does not hold when it meets a cold start.

Recommender-system research has a name for a related problem. A 2015 paper on e-commerce described a continuous cold start, where even known users and items behave as new because visits are rare, interests shift, or the system fails to match a returning user who logs in on another device. The same applies to agents: a customer can look new to the agent while the business knows them well, which is why depth must be measured as the agent sees it.

Define the deployment population before you sample

Start by writing down who the agent will serve: which channels, which customer types, which products, and what share of cases you expect from first-time, occasional and long-standing customers. That expected mix is the target the evaluation is weighted to, and it often differs from the mix in the archive.

Then describe the sampling frame honestly. The Datasheets for Datasets questionnaire asks whether a dataset contains all possible instances or a sample, what the larger set is, whether the sample is representative of it and how that was validated, and whether information is missing from individual instances. Answer those questions for the evaluation set, with history depth as one of the dimensions.

Where the agent falls in a regulated category, the bar is formal. For high-risk systems, Article 10 of the EU AI Act requires training, validation and testing data sets to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose, with appropriate statistical properties for the persons or groups the system is intended to be used with. Most customer-service agents are not high-risk systems, but the test is a sound one either way; counsel decides whether the Act applies.

Measuring history depth at the moment of the case

Compute depth as of the case timestamp, never as of today. A customer with 40 orders now may have had none when the case in your sample was opened, and labeling that case as long-history hides exactly the cold start you want to test. Count only interactions dated before the case and visible to the agent through the tools it will have in production.

Resolve identity the way the production agent will. If the agent sees history only for logged-in customers or exact email matches, a returning customer writing from a new address is a first contact for the agent. Record an identity-match confidence alongside depth, and treat low-confidence matches as their own stratum rather than forcing them into one.

Choose the depth measure that fits the work: prior tickets for support, prior orders for commerce, prior jobs for field service, tenure for subscriptions. Keep the definition fixed for the life of the benchmark so later runs stay comparable.

The customer history depth stratified evaluation design

The table below is the design. Cutoffs and allocations are illustrative starting points; set them from your own distribution so that each stratum is large enough to give a usable estimate.

The customer history depth stratified evaluation design
Design elementDecisionIllustrative defaultWhat to record
PopulationWho the agent serves, by channel and customer typeInbound support email and chat, all customer typesWritten scope and exclusions
Depth strataPrior visible interactions at case timeH0 none; H1 one or two; H2 three to ten; H3 more than tenDepth count and stratum per case
Identity confidenceHow sure the match to prior history isSeparate stratum for low-confidence matchesMatch method and confidence
AllocationCases drawn per stratumEqual counts per stratum, for example 100 eachCounts sampled and available per stratum
Deployment weightsExpected share of each stratum in productionFrom recent intake or launch plansWeights, source and date
Split ruleKeep each customer in one split; test on later periodsGroup by customer id; time-ordered holdoutSplit assignment per customer
Outcome maturityMinimum age before an outcome is finalExclude cases younger than the reopen windowCutoff date and excluded count
ReportingPer-stratum scores plus weighted totalCounts, rates and intervals per stratumBoth weighted and unweighted totals

Splitting the data so familiarity does not leak

Split by customer, not by case. The scikit-learn documentation describes GroupKFold as a variation of k-fold that ensures the same group is not represented in both testing and training sets, and notes that a model flexible enough to learn person-specific features could otherwise fail to generalize to new subjects. Use the customer id as the group, and StratifiedGroupKFold when you also want to keep stratum proportions similar across folds.

Then hold out time. The same documentation warns that classical cross-validation assumes independent samples, and that for time-correlated data it is important to evaluate on future observations; TimeSeriesSplit makes each training set a superset of the ones before it and tests on what follows. For agent benchmarks, the practical version is to tune on earlier months and test on a later period, which also captures new customers who did not exist in the training window.

Check the split after you make it. Count how many evaluation customers also appear in any prompt examples, fine-tuning data or retrieval index the agent uses. Any overlap above zero should be explained or removed.

Worked example: the same agent under two population mixes

All numbers here are illustrative. An evaluation set holds 100 cases in each of four depth strata. The agent resolves 58 of the first-contact cases, 71 in H1, 83 in H2 and 90 in H3.

Weighted to the archive mix, where first contacts are 8 percent of cases, the agent's estimated resolution rate is about 81.8 percent. Weighted to a launch plan where first contacts are 30 percent of incoming cases, the same per-stratum results give about 73.9 percent. Nothing about the agent changed; the population did. The scikit-learn documentation notes that most metric implementations accept a sample_weight parameter, so this reweighting can be computed directly.

Worked example: the same agent under two population mixes
Stratum (illustrative)Resolved of 100Archive mixLaunch mix
H0 first contact588 percent30 percent
H1 one or two prior7117 percent25 percent
H2 three to ten prior8335 percent25 percent
H3 more than ten prior9040 percent20 percent
Weighted resolution rateNot applicableAbout 81.8 percentAbout 73.9 percent

Who is missing from the archive relative to the agent's intended users?

The cases least likely to be in the archive are often the ones the new agent will see most. Look for them deliberately before trusting any stratum's rate.

Report missing groups the way model documentation recommends reporting results. The Model Cards paper proposes breaking evaluation results down by relevant groups and conditions, including intersections of two or more, rather than publishing one average. Depth crossed with channel or product is often where the gaps show.

  • First contacts that never became a record: abandoned chats, unanswered web forms, calls that were not logged.
  • Customers who churned or were deleted under retention rules, which removes long histories that ended badly.
  • Customers from an acquired business or a migrated system, whose history may sit in a separate archive.
  • New channels or regions added after the archive period, which have no depth at all yet.
  • Cases too recent for a final outcome, such as tickets still inside the reopen window or orders inside the return period.
  • Seasonal cohorts, such as holiday first-time buyers, that appear only in some months.

Where depth stratification misleads, and what to do instead

Depth is tangled with case type. New customers ask about onboarding, billing setup and eligibility; long-standing customers ask about renewals and edge cases. A lower H0 score can reflect harder questions rather than missing context, so compare within case types, or add case type as a second stratification dimension, before concluding that history drives the gap.

Some agents never meet a cold start by design. An agent that runs only inside an authenticated account portal sees no true first contacts, and weighting toward H0 would test a population it will not serve. Even then, keep the low-confidence identity stratum, because customers who appear new to the agent still occur.

Small strata limit what you can claim. With 100 cases per stratum, a difference of a few points between strata is within ordinary sampling noise, so report intervals and avoid ranking agents on gaps smaller than the uncertainty. A research statistician should sign off the allocation and weights before the benchmark is used for a buying decision.

Licensing records that cover the cold start

A useful evaluation source includes cases with a customer key, timestamps for every prior interaction, channel, outcome and outcome date, so depth can be computed at case time. Records from a business that grew during the period are especially valuable because they contain many first contacts with known outcomes.

The holder of such records needs the right to license them, which is a separate question from having system access or from de-identifying them, and customer data needs privacy review before any delivery. As a general guide, buyers look for two or more years of consistent history and tens of thousands of records or more in one system, though rare work can qualify with less. An evaluation team that wants cold-start coverage can ask SourceX to source it: datasets come from U.S. businesses on request rather than from stock, and the request form records the scope, volume, format, timing and licensing needs, including the history-depth mix the team wants represented.

Frequently asked questions

What is new customer cold start coverage in a dataset?

It is how well a dataset represents cases where the customer has little or no prior history visible to the system. Measure it by computing history depth at case time and comparing the share of thin-history cases with the share you expect in deployment.

Why do repeat customers inflate agent benchmark scores?

Their cases come with prior tickets, orders and preferences the agent can reuse, and long-standing customers generate many more records than new ones. A random sample from an archive therefore overweights easier, familiar cases.

Should history depth be computed as of today or as of the case?

As of the case. Counting a customer's current history for an old case labels a first contact as a long-history case and hides the cold start you are trying to test.

How do you stop the same customer appearing in training and test data?

Split by customer id rather than by case, using a group-based method such as GroupKFold, and hold out a later time period. Then check for overlap with prompt examples, fine-tuning data and retrieval indexes.

Is reweighting to a deployment mix always right?

Only when you know the deployment mix and each stratum has enough cases. Report per-stratum results and both weighted and unweighted totals, and state where the weights came from and when they were set.

Sources

  • scikit-learn's cross-validation guide says GroupKFold ensures the same group is not represented in both testing and training sets, describes StratifiedGroupKFold, and says TimeSeriesSplit training sets are supersets of earlier ones. Source
  • scikit-learn's model evaluation guide says most metric implementations let each sample provide a weighted contribution through the sample_weight parameter. Source
  • Datasheets for Datasets asks whether a dataset is a sample of a larger set, whether the sample is representative and how that was validated, and whether information is missing from instances. Source
  • Model Cards for Model Reporting proposes reporting evaluation results disaggregated across relevant groups and conditions, including intersections. Source
  • Article 10 of Regulation (EU) 2024/1689 requires high-risk system training, validation and testing data to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete, with appropriate statistical properties for the intended persons or groups. Source
  • The continuous cold start paper describes a recurring cold start in e-commerce affecting even known users, including users who appear new when they do not log in or use another device. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies