Skip to content

AI uses for records

Older records, worse odds? A CRM dataset forecast calibration experiment

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Historical CRM data improves forecast calibration only when a test on later, unseen months shows it, so run a CRM dataset forecast calibration experiment before paying for history: score each probability with the Brier score and log loss against today's forecast on rolling held-out months. In a public replay of 41,188 bank telemarketing contacts, three months of contact history beat last month's rate in all nine 2010 months; the full archive lost in seven.

Record timeline

Older records

This page covers older records, worse odds? A CRM dataset forecast calibration experiment.Illustrative

Key takeaways

  • A shuffled 80/20 split made the replay model look calibrated to within half a point; trained on everything before 2010 and tested on 2010, it forecast 26.9 percent against an observed 52.1 percent.
  • Contact-history fields carried real signal: with a three-month training window they cut the 2010 Brier score from 0.251, last month's rate, to 0.215.
  • History as account fields and history as training rows are different purchases; the full archive as training rows raised the 2010 Brier score to 0.285 until the intercept was refit on the latest month.
  • Economic context fields did not settle the outside-factor explanation: they hurt in 2010 and helped in late 2009.
  • Fix the target, horizon, cutoff, held-out months, baselines, scoring rules, pass rule and outcome codes before the first score.

Does more CRM history make win probabilities more trustworthy?

Not automatically. Older CRM records improve a forecast's calibration only when they still describe the odds a team faces now, and the only way to know is to score forecasts on months the model never saw. A forecast is calibrated when its numbers come true at the stated rate: scikit-learn's guide puts it as roughly 80 percent of the cases given a probability near 0.8 turning out positive.

That turns a vague question into a decision a revenue operations team can test before it licenses, migrates or keeps years of opportunity records for a forecasting model. Does adding the history lower a proper score on later periods, compared with the probability the team already uses? This page sets out that test and runs it on a public dataset, so the mechanics and the trap are both on the table.

The trap is drift. A table can be deep in history and still teach the wrong base rate. In the replay below, the share of contacts that ended in a sale rose from 4.8 percent in 2008 to 52.1 percent in 2010, and a model that leaned on the oldest years forecast far too low.

Three numbers a calibration test must report against the current forecast

Report the Brier score, the log loss and calibration-in-the-large, each for the new forecast and for the forecast it would replace. The Brier score is the mean squared gap between the stated probability and the outcome coded 1 or 0; scikit-learn's documentation calls it equivalent to mean squared error and quotes its description as a strictly proper scoring rule. Log loss punishes confident misses harder, which matters when a model says 2 percent and the deal closes.

Calibration-in-the-large is the mean forecast minus the observed rate over the test period, a fast read on whether the whole forecast sits too high or too low. Murphy's 1973 partition of the probability score splits the Brier score into three terms: the uncertainty of the events, the reliability of the forecasts and their resolution. Reliability is the calibration part; resolution is how well the forecast separates likely deals from unlikely ones; uncertainty is set by the base rate, and no model can change it.

The comparison runs against the forecast in use, not against zero. Salesforce's Opportunity object, for example, carries a Probability percent field beside a Forecast Category; whatever fills that field, a stage default or a rep's estimate, is the probability to beat. The replay has no stages, so its stand-in is last month's observed rate.

Why a shuffled split flatters a forecast that drift will break

A random split mixes old and new records into both the training and the test set, so the model is graded on the same blend of eras it learned from. A chronological holdout trains on the past and scores the future, which is how the forecast will be used. Forecasting: Principles and Practice calls the repeated version evaluation on a rolling forecasting origin, where the origin moves forward and accuracy is averaged over the test sets.

The replay shows the size of the gap. Shuffled with a published seed (20261027) and split 80/20, the contact-history model scored a Brier score of 0.086, with a mean forecast within half a point of the observed rate; ten further seeds gave Brier scores from 0.084 to 0.091. Trained on every month before 2010 and tested on the 2,058 contacts of 2010, the same model forecast 26.9 percent on average against an observed 52.1 percent, a Brier score of 0.291. Last month's rate alone scored 0.251 over those months.

The shuffled test was not wrong about the data it saw. It answered a question nobody asks of a forecast, which is how well it describes a pooled past. Any calibration claim made for historical CRM data should name the split that produced it.

A CRM dataset forecast calibration experiment on 41,188 public contacts

The data are the UCI Bank Marketing records from a Portuguese bank's phone campaigns: 41,188 contacts with 20 inputs, which UCI describes as ordered by date from May 2008 to November 2010. The copy used here is the one bundled in the R package nlpred 1.0.1 on CRAN, which keeps the published row order. It stands in for an opportunity table: each row is a sales contact, the outcome is whether the client subscribed to a term deposit, and two fields record earlier campaigns.

Inputs were limited to what a seller knows before the call. Current-campaign fields: contacts made in this campaign and whether the contact was by telephone rather than cellular. History fields: contacts made before this campaign and the previous campaign's outcome. Call duration was left out because UCI notes it is not known before a call is performed yet strongly affects the outcome, and no demographic field was used.

Each test month was forecast only from earlier months, with the training window set to the last 3, 6 or 12 months holding data, or to all of them. The models are logistic regressions with a fixed penalty, so every run gives the same numbers. Two blocks were scored: the nine months of 2010 and June to December 2009. Main results for 2010, with late 2009 beside them:

A CRM dataset forecast calibration experiment on 41,188 public contacts
Forecast2010 Brier score2010 mean forecast (52.1% observed)2010 months better than last month's rateJun-Dec 2009 Brier score
Pooled rate of all prior months0.42610.1%0 of 90.336
Last month's rate (the baseline)0.25152.3%Reference0.256
Current-campaign fields, last 3 months0.24052.8%5 of 90.267
Adding contact history, last 3 months0.21553.3%9 of 90.248
Adding contact history, all prior months0.28529.3%2 of 90.284
All prior months, intercept refit on latest month0.23153.0%7 of 90.245
Contact history plus economic context, last 3 months0.22760.9%8 of 90.246

Three months beat the whole archive: reading the window test

In 2010, contact history earned its place. Trained on the last three months, the history model scored 0.215 against 0.251 for last month's rate, a Brier score 14.4 percent lower, and it won in all nine months. The same model without the history fields scored 0.240, so most of the gain came from knowing how a client had answered a previous campaign.

Lengthening the window reversed the result. With 12 months of training the Brier score was 0.249, and with every prior month it was 0.285, worse than last month's rate in seven of nine months, because the older contacts carried a far lower success rate. The model still ranked contacts sensibly; it had learned the wrong level.

Refitting only the intercept on the most recent month, with the other coefficients learned from the earlier months of the window, repaired most of the damage: 0.222 at 12 months and 0.231 with all history. The latest month is kept out of the slope fit because, as scikit-learn's guide puts it, a calibrator is ideally fit on data independent of what trained the classifier. The step has a cost at short windows: with three months, it left two for the slopes and scored 0.220, slightly worse than 0.215 without it. Late 2009 told the same story by smaller margins, with three months of history at 0.248, last month's rate at 0.256 and all history at 0.284.

For anyone buying or keeping CRM history, the replay separates two uses of old records. As fields, they describe each account, and a previous outcome exists only if earlier records were kept. As training rows, they set the base rate, and stale rows set it wrong unless a held-out test shows otherwise.

Dot chart of the 2010 Brier score by training window: the raw contact-history model rises from 0.215 at three months to 0.285 with all prior months, the intercept-refit version stays between 0.218 and 0.231, and last month's rate sits at 0.251.
Public replay of UCI Bank Marketing data (nlpred 1.0.1); lower is better; values from the article's script.

Was it the economy rather than the records? Testing the outside-factor explanation

A won deal can owe more to the market than to anything in the CRM, so the strongest alternative explanation is that outside conditions, not contact history, moved the odds. The UCI file carries economic indicators, so a further variant added three of them to the history model: the three-month Euribor rate, the employment variation rate and the number employed.

The result did not settle it. In 2010 the context fields made the three-month model worse, 0.227 against 0.215, and pushed its mean forecast to 60.9 percent against 52.1 percent observed. In late 2009 the all-history model with context scored 0.234, the best figure in that block. Economic indicators shift with the calendar, so they largely mark when a record was made, and a model can lean on them as a clock rather than as an explanation.

A calibration score also cannot say whether any single win was deserved. It grades the probability, not the choices made along the way, so a well calibrated forecast can sit on top of poor intermediate decisions. Keep decision quality as its own label, built from its own evidence.

The calibration test card to fix before scoring

Write each element down before the first run, so nobody picks the window or the baseline after seeing which one wins. Each row below gives the replay's setting for one element of the card:

The calibration test card to fix before scoring
ElementWhat to fix in advanceSetting in the public replayMistake it prevents
Target and horizonThe event scored and the window it must happen in, such as closed-won within 90 days of the cutoffContact ended in a purchaseScoring deals against different deadlines
As-of cutoffEvery input as it stood at the cutoff, rebuilt from field history rather than today's recordInputs known before the call; duration excludedLeaking the outcome into the inputs
Held-out periodsCalendar months after the training data, rolled forward one at a timeNine months of 2010 and seven of late 2009A shuffled split that blends eras
BaselinesThe probability in use today plus a recent observed rateLast month's rate and the pooled rateBeating a straw man
Scoring rulesBrier score and log loss, with calibration-in-the-large and the reliability and resolution termsAll four, pooled and by monthReporting accuracy or one chart
Pass ruleA Brier improvement that holds in most held-out months, set before the runMonths better than last month's rateDeclaring victory on one good month
Outcome codesHow won, lost, slipped and withdrawn records count at the horizonNot needed: every call resolvedQuietly dropping awkward deals

Horizon outcome codes for opportunities that refuse to resolve neatly

Opportunity data brings a complication the replay lacked: deals take months, and many are neither won nor lost when the horizon arrives. Give every record a status code at the horizon before scoring and publish the count in each code, so a forecast cannot look calibrated because difficult deals disappeared. A workable set of codes:

  • Won inside the horizon: scored 1.
  • Closed lost inside the horizon: scored 0, with the loss reason kept in a separate field.
  • Open past the horizon: scored 0 for this horizon, since the forecast said it would close in time; its eventual result answers a later question.
  • Withdrawn before the horizon: scored 0 when it was in the forecast population at the cutoff, never deleted.
  • Not yet scoreable: the horizon has not passed; held back and counted, never imputed.
  • Removed before scoring: duplicates and test records, reported by count with the rule that removed them.

Putting the experiment into a request for CRM records

A buyer can ask for records that make this test possible without asking for anyone's contact details. SourceX sources datasets on request from U.S. businesses rather than holding stock, and its request form is where a buyer spells out what it needs, from scope and volume to format, timing and license terms; the test card belongs in that request. Describe what the experiment needs: opportunity or contact records with outcome dates, field history that lets each record be rebuilt as of a cutoff, enough months to hold out at least two calendar blocks, and the stage or probability the team forecast with at the time.

As a general guide, SourceX notes that buyers look for two or more years of consistent history in one system, assessed deal by deal. The replay suggests how to spend those years: as account fields and as held-out test periods, with the training window chosen by the test. Make the card part of acceptance, so the records pass when the buyer can rebuild as-of snapshots and run every element, not when a model improves, and leave room for the answer to be no. Names, emails and phone numbers serve no element on the card; account identifiers that stay stable over time are enough.

What the replay cannot show, and the evidence to collect next

The replay covers one bank's phone campaigns, not a B2B pipeline. Each outcome was known the day of the call, there were no stages or rep forecasts to beat, and the rise in the success rate came with a steep fall in volume, from 27,690 contacts in 2008 to 2,058 in 2010, so who was called probably changed along with the odds. Several test months hold fewer than 250 contacts, months won is a coarse measure of consistency, and no confidence intervals are reported.

Methods: the file data/bank.RData from the GitHub mirror of nlpred at tag 1.0.1 (commit ba39efa, SHA-256 b2d31e93), read without running package code. The year is rebuilt by adding one whenever the month number falls between consecutive rows, which yields 26 months from May 2008 to November 2010. Inputs are the log of contacts in this campaign, the telephone flag, the log of one plus earlier contacts, and flags for a previous success and for no previous campaign; pdays, duration and every demographic field are excluded, and the context variant adds the three indicators, standardized on its training rows. Logistic regression is fit by Newton's method, minimizing summed log loss plus 0.5 times the squared coefficients (every coefficient except the intercept). In the shuffled comparison, rows are permuted with numpy default_rng(20261027) and the first 80 percent are used for training. Log loss clips probabilities at 1e-12; the Murphy terms use ten equal-width bins; months won compares unrounded monthly Brier scores; values are rounded once, at the end. Reviewers can rerun the replay from the saved script.

The next evidence is the same card run on real opportunity history: as-of snapshots, deal outcomes coded at a fixed horizon, the stage or rep probability as the baseline and at least two separate held-out years. If three months of training keep beating the full archive there too, most of the value of old records sits in the account fields they create, and a forecasting team should price them that way.

Frequently asked questions

Why not test calibration with a random train-test split?

A random split puts records from every period into both halves, so it grades the forecast on the blend of eras it learned from. A forecast is used on future periods, so it should be tested on them. In the public replay, a shuffled split showed a mean forecast within half a point of the observed rate, while training before 2010 and testing on 2010 missed by about 25 points.

How many held-out months does a CRM calibration experiment need?

Enough to see whether a gain repeats, not just whether it appears once. Hold out at least two separate calendar blocks, report the pooled score and the number of months the new forecast beat the baseline, and merge thin months into quarters when deal counts are small. The replay used nine months of 2010 and seven months of 2009.

Should old CRM records be dropped if they hurt calibration?

Usually not. Old records create account fields, such as how a client answered last time, that a model cannot get any other way. What the test should decide is the training window and whether the intercept needs refitting on recent data before forecasts are used.

What does refitting the intercept change in a forecast?

It shifts every probability by the same amount on the log-odds scale so the average matches a recent period, without changing how deals are ranked against each other. Fit it on data the model was not trained on. In the replay it cut the all-history model's 2010 Brier score from 0.285 to 0.231.

Can open deals be included in a calibration test?

Only once their horizon has passed. Before that they are not yet scoreable and should be held back and counted. After it, a deal still open counts as not closed within the horizon, because that is what the forecast claimed would happen.

Sources

  • The nlpred 1.0.1 copy of the UCI Bank Marketing data: 41,188 contacts in published date order, the file the replay reads; all replay figures come from it. Source
  • The nlpred documentation reproduces the UCI description of the bank data, including the date ordering and the warning about call duration. Source
  • UCI describes bank-additional-full as 41,188 examples with 20 inputs ordered by date from May 2008 to November 2010, and says call duration is not known before a call and should be dropped from realistic models. Source
  • Murphy's 1973 paper partitions the probability score into the uncertainty of the events, the reliability of the forecasts and a measure of resolution. Source
  • scikit-learn documents the Brier score as equivalent to mean squared error and quotes its description as a strictly proper scoring rule. Source
  • scikit-learn's calibration guide defines a well calibrated classifier and says a calibrator is ideally fit on data independent of the classifier's training data. Source
  • Forecasting: Principles and Practice describes time series cross-validation as evaluation on a rolling forecasting origin, averaging accuracy over test sets. Source
  • Salesforce's Opportunity field reference lists a Probability percent field and a Forecast Category picklist. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies