1
$\begingroup$

I'm working on a model to predict churn. I understand the concept of training and testing, or at least I thought I did.

Let's say it's the first of the month and our database has 10,000 subscribers, in addition to 10,000 who have cancelled. I now have a dataset of 20k records. If I wanted to build a model to predict which of these 10K active subscribers are likely to cancel, with a view to sending them an email with e.g. a discount offer code, how would I split my data between training and testing, and then for prediction?

Ideas:

  1. Take the entire dataset of 20K, split 80% train and 20% test. But then, what do I predict on? Do I want to use all data to get best training since I'll have more records?
  2. Take a sample of say 10k records from the total dataset, test and train with 0.8 and 0.2 as above and then predict on the whole data set? But then that would mean I'm predicting on training data? That must be "wrong" surely?

This made sense in class. Now that I'm trying to apply it to real world I'm a little lost.

Of the 20k records, what would be a "traditional" way of splitting it up between test and training data, and then which data for prediction?

Improve this question
$\endgroup$
2
  • $\begingroup$ Observe that for 10k of your records, you do not have an observed response to compare your predictions to. How does that inform your thinking? $\endgroup$
    – Matthew Drury
    Sep 30, 2015 at 14:24
  • $\begingroup$ Hi @MatthewDrury. Well, on the first of the month when I pull the data I use if logic: if cancelled label is "churned" else if not cancelled label is "not churned". Is my question clear? If at month start boss asks "Who of our current users are likely to churn as of just now", how would you split test/train/predict? $\endgroup$
    – Doug Fir
    Sep 30, 2015 at 14:52

2 Answers 2

Reset to default
1
$\begingroup$

Call your target churned/not churned, and represent it as a binary output. Now you have 20k lines, half of them being 0, the other half, 1. All the other data that you have can be considered as predictors.

Now, from this 20k lines, randomly sample say 5k of them and keep the 15k others as your train set.

You can train your model : churned=f(predictors) on the 15k lines (don't use the remaining 5k lines now) and try different models on it. For each model you train on this set, you can evaluate its performance on the test set.

For the best model you found on the train set (or the best pair model/hyperparameters), train it on the complete set (the 20k lines).

The obtained model can be used in production and, hopefully, will have the same performances than those observed on the test set.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer. I think I've failed to communicate my question properly. The keyword that you used there is "production". I want to train, test AND predict on this 20k. So put another way, how should I separate production data (for prediction) from train & test data? Of the 10k active users we want to know who are likely to churn. So because it's the first of the month, there's no new production data, just the whole thing. Make sense? $\endgroup$
    – Doug Fir
    Sep 30, 2015 at 16:00
  • $\begingroup$ I need to edit that slightly, allow me to put another way again. It's the 1sdt of the month and we have 10k active users, as well as records on 10k deleted accounts. If I want to predict how many of these active 10k are likely to churn, how would I break down my total data set into train, test and predict? Predict presumably has to be on 10k active users since it's from those people we want to know who is going to churn. $\endgroup$
    – Doug Fir
    Sep 30, 2015 at 17:46
  • $\begingroup$ If I understand you correctly, there are 10 k users that are still there, and you wish to know which one are the most likely to leave, given that some users left... I am not good at these things. I would probably go for an unsupervised approach and look at the response cluster by cluster... $\endgroup$
    – RUser4512
    Sep 30, 2015 at 20:52
0
$\begingroup$

What if you ramdonly split your dataset into two? Now you have 2 datasets each with 10K customers. Each dataset in turn has a mix of both churned and active customers. You can then use the first dataset, split into train/test, create a model and then use that model to predict active customers in second dataset. Curious what others think of this approach.

PS: I've asked a similar question but related to survival models. Splitting between train/test for customer churn survival models

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.