Systems and records
Renewal decision context dataset: half a churn signal was contract mix
By SourceX Editorial · Updated
8 sections
Short answer
A renewal decision context dataset should hold one row per renewal decision opportunity: the decision event, who held authority, the options and save offers made, and support, usage and billing context joined as of the decision. Treat every signal as an association until tested. In IBM's public Telco sample, contract type alone explained about half the churn gap linked to a tech-support add-on.
License terms
Renewal decision context
Key takeaways
- Build renewal datasets around decision opportunities, not customers, and count accounts that had no real option to leave.
- Silent renewals, involuntary lapses and structural exits are not decisions and need their own labels.
- In IBM's public Telco sample, a 26.5-point churn gap tied to a support add-on fell to 12.6 points within contract types.
- Support and usage signals imitate causes through structure, timing, intervention and revision, and each has a test.
- Join context as of the decision event and filter on when each value was recorded, not only on the date it describes.
What belongs in a renewal decision context dataset?
One row per renewal decision opportunity, not per customer: the moment a customer could renew, change terms or leave, who had authority to decide, the options on the table, anything the vendor did to sway the choice, the context available before the decision, and the outcome. Support cases and usage logs are part of that context, but only one part, and none of them is a cause until a design shows it.
The obvious build goes wrong in a predictable way. Join every support case, usage metric and billing event to a list of renewed and churned accounts, train a model, and it will find strong patterns: more tickets before churn, falling logins before cancellation, discounts before renewal. Some of those patterns are causes. Others are the decision showing up in the records after it was made, the vendor's own interventions, or differences between kinds of customers that have nothing to do with service.
This page sets out the decision unit and a label taxonomy, a public recount of how much of one churn signal disappears once contract structure is held fixed, the four ways context records imitate causes, the join rules that keep post-decision evidence out, and a design for testing a cause with permissioned records.
What counts as a renewal decision, and who makes it?
A renewal decision exists when a customer has a real option to renew, change terms or leave, and someone with authority exercises it, actively or by letting a default run. A customer midway through a multi-year term usually has no such option in that period, so it belongs outside the decision set, counted but not labeled.
Authority shapes the record. The people who file support cases and generate usage are often not the people who sign renewals, and the economic buyer may never appear in a ticket. Record the role that decided and the roles whose activity makes up the context, so a model does not learn that the users' mood is the buyer's decision.
Outcomes need more than renewed or churned. A silent auto-renewal carries no decision evidence at all, a renewal bought with a concession differs from a willing one, and an account lost to a failed payment or an acquisition did not choose to leave. Billing systems keep these paths apart: Kill Bill's documentation, for example, describes overdue or dunning flows that configure how the system behaves when an account is overdue, a route to an ending that no customer decided. The label taxonomy, with the evidence each label needs:
| Label | What happened | Evidence required | Use in training |
|---|---|---|---|
| Active renewal | Authorized renewal on the same or better terms | Signed order or renewal approval dated before the term ended | Positive outcome |
| Renewal on concession | Renewed after a discount, credit or term change offered to keep it | Offer record with date, approver and size as a percentage | Positive outcome, intervention flagged |
| Silent renewal | Default auto-renewal with no decision evidence | Renewal clause and billing event, no customer action | Its own class, kept out of decision labels |
| Contraction | Renewed with fewer seats, modules or volume | Entitlement records before and after | Partial outcome, measured as change |
| Expansion | Renewed with more scope | Entitlement records before and after | Positive outcome, scope change recorded |
| Voluntary exit | Authorized decision not to renew | Notice or cancellation request from an authorized role, with its date | Negative outcome |
| Involuntary lapse | Service ended through failed payment or dunning | Billing and overdue records with no cancellation request | Excluded from decision labels |
| Structural exit | Customer acquired, merged or closed | Dated corporate event before the decision | Excluded, or coded as an outside cause |
A public recount: how much of one support signal was contract mix?
About half of it. In IBM's public Telco customer churn sample, customers without a tech-support add-on churned at 41.6 percent against 15.2 percent for those with it, a 26.5-point gap. Within contract types the gap was 19.7 points for month-to-month customers, 1.1 for one-year and 2.4 for two-year contracts; weighted by contract mix it falls to 12.6 points, 48 percent of the raw figure.
The sample is useful because it is public and widely reused, not because it describes a real business. IBM's Cognos Analytics community describes the data as a fictional telco that provided home phone and internet service to 7,043 California customers, with a churn flag for customers who left within the last month. The copy we recounted, from IBM's telco-customer-churn-on-icp4d repository at commit d5371f5, has 7,043 rows and 21 columns, and no column name contains a date or a time.
Contract structure dominates the outcome. Month-to-month customers churned at 42.7 percent, one-year customers at 11.3 percent and two-year customers at 2.8 percent, and month-to-month customers make up 1,655 of the 1,869 churners, 88.6 percent. They also hold the add-on far less often: 20.0 percent of internet customers on month-to-month terms have it, against 49.8 percent on one-year and 77.7 percent on two-year contracts. A pooled comparison therefore sets add-on holders, mostly on long contracts, against non-holders, mostly month-to-month.
This is the pattern the Stanford Encyclopedia of Philosophy describes as Simpson's paradox: an association between two variables in a population can emerge, disappear or reverse when the population is divided into subpopulations. Here it shrinks rather than reverses. Read it with care: the add-on is a product holding, not a ticket count; the file has no dates, so nothing can be ordered in time; and even the month-to-month gap is an association, because customers who buy support may differ in ways the file does not record. The recount, by contract type:
| Contract type (internet customers) | Customers | Hold the add-on | Churn with add-on | Churn without add-on |
|---|---|---|---|---|
| Month-to-month | 3,351 | 20.0% | 30.7% | 50.4% |
| One year | 1,109 | 49.8% | 13.6% | 14.7% |
| Two year | 1,057 | 77.7% | 3.5% | 5.9% |
| All internet customers, pooled | 5,517 | 37.0% | 15.2% | 41.6% |
Four ways support and usage records imitate a cause
Through structure, timing, intervention and revision. Each produces a real correlation between a context signal and the renewal outcome that would not survive a change in the signal itself, and each has a test that can be built into the dataset.
Intervention deserves the most care, because the vendor creates it. Customer success teams aim attention, discounts and executive calls at accounts they believe are at risk, so treated accounts differed before any treatment began. Eva Ascarza's study in the Journal of Marketing Research (2018), which combined two field experiments with machine learning, found that the customers most likely to churn are not necessarily the best targets for a retention program, and that targeting customers by their response to the intervention was significantly more effective than targeting by risk. A dataset that logs interventions without logging who was eligible and who went untreated cannot tell a working save program from a well-aimed one.
The four patterns, with the test and the field each one needs:
| Pattern | How it looks in the data | Non-causal explanation | Test and field required |
|---|---|---|---|
| Structure | Accounts with a support add-on or more tickets renew at different rates | Contract length, size, segment or tenure drives both | Compare within contract type and size band; keep term length and start date |
| Timing | Usage falls and tickets spike before cancellation | The decision came first and the activity followed it | Anchor at the decision event and drop later activity; keep decision and notice timestamps |
| Intervention | Saved accounts received discounts or executive calls | Help went to accounts already judged at risk or likely to stay | Log eligibility, assignment rule and untreated comparisons; keep offer dates and approvers |
| Revision | Severity or usage values line up suspiciously well with outcomes | Fields were corrected or backfilled after the outcome was known | Filter on when each value was recorded; keep event and created timestamps |
How to join context as of the decision, not as of today
Anchor every join at the decision event and keep only values that existed, and had been recorded, before it. That takes a decision timestamp on every row, a lookback window chosen in advance, and a filter on when each value entered the system, not only on when the event it describes took place.
At a glance
How to join context as of the decision, not as of today
Feature-store software makes the distinction explicit. Feast's documentation describes a point-in-time join that scans backward from each row's timestamp up to a time-to-live window, and warns that by default it constrains only the event timestamp, so a value backfilled or corrected after a row's timestamp can still be returned unless retrieval filters on the created timestamp. Renewal data carries the same trap in any system where staff can reclassify a ticket's severity or rerun a usage job.
The anchor needs the same care, because billing systems separate dates that look alike. Kill Bill's documentation explains that a cancellation acts on entitlement and on billing, which are connected but not necessarily aligned: a customer who cancels immediately can lose access at once while billing runs to the end of the period already invoiced. For a decision dataset, the request is the event, and the access end and billing end are consequences. Join rules for each decision row:
Checklist · 6 items
- Store the decision event time, the term end and the effective end as separate fields, and anchor joins at the decision event.
- Fix the lookback window before looking at outcomes, for instance an illustrative 90 days of usage and 180 days of support history, and record it with the data.
- Exclude any record created after the decision event, even when it describes an earlier date.
- Keep an eligibility flag, and count the rows excluded because the customer had no real renewal option in the period.
- Log every intervention with its date, approver and assignment rule, including eligible accounts that were left untreated.
- Keep roles, not people: the deciding role, the requesting role and the roles behind support and usage activity.
0 of 6 ticked. Tick each item as you go.
Could a renewal be won for reasons the records never show?
Often, and the dataset should leave room for it. A customer may renew because switching costs are high, a budget cycle favors keeping what exists or a rival stumbled, even with a poor support history; a customer may leave despite excellent service because it was acquired or changed strategy. A renewal outcome also rewards everything that came before it, including poor intermediate decisions that happened not to matter.
Two fields keep service quality from being read into every outcome. An outside-cause code, tied to dated evidence such as an acquisition or a budget freeze, separates structural exits from service exits. An intermediate-decision log, covering escalations handled, promises made and offers approved, lets a reviewer judge those decisions on their own evidence instead of letting each one inherit the renewal result.
Both fields make the dataset more honest about what it cannot explain. If a large share of exits carry outside-cause codes, a model trained to predict churn from service signals is being asked to explain events that service never touched, and its apparent accuracy on the rest should be read in that light.
Testing a cause with permissioned records: the design, results not yet published
Use the renewal records to find candidate causes, then test them with a design that can fail. SourceX has not run this study on any company's records; the design is set out so that a vendor or a data buyer can, and results are not yet published. The design:
At a glance
Testing a cause with permissioned records: the design, results not yet published
- Population: every renewal decision opportunity in a fixed window, with silent renewals, involuntary lapses and structural exits counted and set aside.
- Candidate signals: support, usage and billing measures named in advance, each computed only from records created before the decision event.
- Comparison: within contract type, size band and tenure band, reporting the pooled and adjusted differences side by side, as in the public recount.
- Interventions: where a save program exists, a random holdout of eligible accounts, assigned with a seed fixed and published before assignment, so the program's own effect can be estimated.
- Uncertainty: a 95 percent interval on every difference, with the number of decisions behind each.
- Stop rule: a signal whose adjusted difference cannot be told apart from zero is reported as such and never used as a causal label.
Specifying a renewal decision dataset in a data request
Ask for decision rows, not account snapshots. Name the unit, the label taxonomy, the join anchor and the intervention log, and attach an acceptance test, so a supplier can say whether its systems hold the timestamps the design depends on.
SourceX keeps no renewal datasets in inventory; each one is sourced from a U.S. business to fit what a buyer sets out about scope, volume, language, format, timing and licensing. Whether a given company's CRM, billing and support systems keep decision and created timestamps is checked on a sample before anyone relies on them. Lines worth writing into the request:
- Unit: one row per renewal decision opportunity, with the eligibility rule stated.
- Labels: the eight-label taxonomy, with silent renewals and involuntary lapses kept apart.
- Anchor: decision event, term end and effective end as separate timestamps.
- Context: support, usage and billing records with event and created timestamps, joined over a fixed lookback.
- Interventions: offers and outreach with dates, approvers and assignment rules, plus untreated eligible accounts.
- Acceptance test: a sample on which pooled and within-contract comparisons can be recomputed from the delivered files.
Frequently asked questions
Do support tickets predict SaaS renewals?
They often correlate with renewal outcomes, but a correlation can come from contract structure, the timing of a decision already made, the vendor's own save efforts or values revised after the fact. In IBM's public Telco sample, about half of the churn gap tied to a support add-on disappeared once customers were compared within contract types.
What is a renewal decision opportunity?
It is a period in which a customer could actually renew, change terms or leave, such as the run-up to a term end or a month-to-month billing cycle. Customers midway through a multi-year term usually have no such option, so they are counted but kept out of renewal labels for that period.
How should silent auto-renewals be labeled in a dataset?
Give them their own label rather than calling them renewals. A contract that renewed by default with no customer action carries no evidence about the customer's choice, so mixing these rows with active renewals teaches a model that inaction is approval.
Why exclude involuntary churn from renewal decision labels?
Because nobody decided to leave. Accounts that lapse through failed payments or dunning flows ended by a billing process, and customers that were acquired or closed ended for structural reasons. Keep both in the data with their own codes, but out of the labels a model uses to learn why customers choose to go.
What does point-in-time correct mean for renewal data?
Every context value joined to a renewal row must have existed and been recorded before that row's decision event. Feast's documentation, for example, warns that a value backfilled after a row's timestamp can still be returned unless retrieval also filters on the created timestamp.
Sources
- IBM's Telco customer churn CSV at commit d5371f5 has 7,043 rows and 21 columns, none named as a date or time; the SourceX recount finds a 26.5-point churn gap tied to the tech-support add-on, 12.6 points after weighting by contract mix. Source
- IBM's Cognos Analytics community describes the Telco churn sample as a fictional company providing home phone and internet service to 7,043 California customers, with churn flagged for customers who left within the last month. Source
- The Stanford Encyclopedia of Philosophy defines Simpson's paradox as an association between two variables in a population that emerges, disappears or reverses when the population is divided into subpopulations. Source
- Ascarza (Journal of Marketing Research, 2018) combined two field experiments with machine learning and found that the customers most likely to churn are not necessarily the best targets for retention programs; targeting by response to the intervention worked significantly better. Source
- Feast's documentation describes point-in-time joins that scan backward from each row's timestamp up to a TTL, and warns that by default a value backfilled or corrected after that timestamp can still be returned unless filtered by created timestamp. Source
- Kill Bill's documentation explains that a cancellation affects entitlement and billing, which are connected but not necessarily aligned; an immediate cancellation can end access at once while billing runs to the charged-through date. Source
- Kill Bill's Kaui guide describes overdue (dunning) flows that configure how the system behaves when an account is overdue. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.