I need to determine a statistically sound methodology for creating training and validation datasets for a churn model. Testing sets and model selection aren't a problem.

The data spans 4 years of business trading. Customers have 12 month contracts. When contracts are up for renewal customers either renew for 12 months or churn. Customers start contracts at different times of the year. Renewal always lags contract inception by 12 months.

The aim is to use a classifier to produce a churn probability score for each customer requiring renewal in an upcoming 3 month period, regardless of their contract year (tenure). The scoring would occur for a new batch of customers every 3 months.

Here's a made up example of what a scoring batch could look like. Say between Jun and Aug 2018 the following customers are up for renewal: 3000 1st yr customers, 2000 2nd yr customers, 1500 3rd yr customers.

Below are the best options I can think of. However I don't have enough knowledge to know which makes the most sense.

  1. Create training and validation sets using all 4 years of data. Score each batch of customers requiring renewal in the upcoming 3 month period against this model. Add 3 months of data to the main training set every 3 months.
  2. Create separate models for all customers who churned historically at year 1, year 2, year 3, etc. Separate each batch of customers to score based on tenure. Score 1st year contract holders against year 1 model, 2nd year contract holders against year 2 model, etc.
  3. Use a rolling window. For example, train on the last 12 months of data. Score on customers requiring renewal in the upcoming 3 month period. Then shift both the training and scoring along 3 months every 3 months.

Which is the most suitable option? Or is there a better approach?

  • $\begingroup$ This question is probably not about general train-valid split methods and is more domain specific, and it depends on the amount of data you have and how similar the 1st year customers are to the 2nd year customers. $\endgroup$ – DiveIntoML Mar 9 '18 at 20:15
  • $\begingroup$ Thanks. If I said the customers are extremely similar and data is circa 1m observations per year, split 70%-30% between the classes, could you offer any more advice? I agree there's some domain specificity here, but also imagine there could be some rule-of-thumb guidelines or 'what I would do given incomplete information' approaches. $\endgroup$ – ls22 Mar 9 '18 at 20:41
  • 1
    $\begingroup$ A Bayesian approach for lifelines can be interesting for your blog.yhat.com/posts/estimating-user-lifetimes-with-pymc.html $\endgroup$ – Vladislavs Dovgalecs Mar 9 '18 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.