I'm trying to create a model to predict churn in the insurance industry. The objective will be to ' predict the probability of each member that will churn next month' i created a one row per member per month dataset where every member has one row for every month he has been active, the demographical information during that month, the household info, the claims made that month, the premiums paid and the calls made to the customer contact center. Below is the sample of what the dataset might look like

Id Year Month product.x product.y claim prem calls

x 2017 8 1 0 250 180 1

x 2017 9 1 0 270 180 0

x 2017 10 1 0 0 180 0

x 2017 11 1 0 250 180 0

Now to the question: I'm trying to predict who is going to churn next month. the test dataset is the active members at the end of this month. the training dataset is the rest of the data, which also has the same members + few members that churned. Since the members in the test dataset are also present in train dataset without any variation across months( in the insurance industry, there is no major change in the activity of individual across months. they pay the same premium as last month, most buy regular drugs-hence regular claims and the demographics change rarely), the model always results that all members will be active.

How do I go about solving this issue? Am I making a mistake in the split of dataset?

Any thoughts will be highly appreciated

update for mkl:

you're right. the data is far from perfect. but we are getting there by incorporating more variables. Currently, I tried the following: • Performed k fold cross-validation on the entire dataset with tree ensemble learner and random forest (predicts all customers to be active. Zero kappa score) • I made a split on the one row per member per month dataset in such a way that I have a designated test dataset consisting of all the members that are active at the beginning of the month. The remaining is the training dataset. So, the training dataset contains the termed members and each member has multiple rows for every month till they terminated. Now the training dataset is quite small (the terminated members in a year are very few compared to the active members in the training dataset). Did oversampling of train data and used GBM. Still no results

  • $\begingroup$ Based on the comments that have emerged, I think that the time-scale of analysis might be potentially inappropriate. I fully appreciate what you say about potentially low information in your sample but that might mean you need to reassess the question you are asking. Something like "zero kappa" is so bad that it is almost unusual and hints to the irrelevancy of a train/test dataset rather than difficulty of prediction. $\endgroup$ – usεr11852 Jul 19 '18 at 20:12

Don't predict yes/no. Instead, predict a probability that someone will churn. Then base any actions (e.g., calling them, offering a discount, whatever) on that probability and the costs of correct/wrong actions.

This earlier thread may be helpful.

  • 1
    $\begingroup$ well I can aim at predicting probability. when the ML algorithm tends to encounter the same row in the test dataset as in the training dataset, it ends up predicting that the customer would stay because there are 24 instances of the same customer record in the train dataset that states he is active $\endgroup$ – k_mohan Jul 19 '18 at 16:52
  • 1
    $\begingroup$ Are you sure that is an issue? If you have the customer represented 20 times for different periods (where they churned in the final period), and the model says there is a 1/20th probability that they will leave, then the model was as right as it can be. $\endgroup$ – Matthew Drury Jul 19 '18 at 18:52

The questions is whether the data you collected can explain churn at all. The model can learn some weights / thresholds but it will not extract knowledge / rules that are outside your data. E.g. if the most influential factor is (change of) income and you don't have it in your data, your predictions will be far from perfect.

Assuming positive answer - i.e. that observing your features can give you a reasonable answer about probability of the churn - how do you predict the churn? What is your model / approach?

Do you use some aggregates over the features and their combinations, like 'sum of claims > sum of premiums over last 6 months', 'number of calls is increasing in every month through last 3 months'?

  • $\begingroup$ please see updated que for answer. ive added some aggregates over features such as call increase, percentage of claims paid as opposed to overall claimed amt $\endgroup$ – k_mohan Jul 19 '18 at 17:50
  • $\begingroup$ Any progress? Sorry for silence, holiday break :) With your update it's a bit clearer what is the problem. So starting from prediction moment (after training a model that I will discuss in a sec), you will have some past data about all current and past customers (a number of months) and you want to predict the status of the current ones in next month. So the features to be used needs to cover a number of past months (or somehow all of them, for each customer varying number). There is a number of ways to handle varying number of features: fixing 'look behind window', aggregates, RNNs. [tbc] $\endgroup$ – MkL Aug 8 '18 at 20:56
  • $\begingroup$ Now when talking about training, obviously we need to use the same features. So for a given customer one data point will be e.g. 10 month worth of the oldest data plus status in 11th month. Then next data point will be 11 month worth of data plus status in 12th (or months 2-11 if you decide for fixed look behind window). You can stop it anywhere you choose and use rest of data for testing, again shifting window appropriatel - this way creating a split train/test. $\endgroup$ – MkL Aug 8 '18 at 20:59

Why don't you assign observations to Train and Test based on ID?

create a column that is ID mod 10, then use if/then logic to assign. Then you won't have the same policyholder in both data sets.

I would also add I would define a fold column based on ID. You've likely got the same person in all of your folds.

Is this insurance bought through an employer or is it direct? Are there variations by time of year?

  • $\begingroup$ thank you! I did a subject wise split already. poor prediction! features dont seem to be strong enough $\endgroup$ – k_mohan Jul 20 '18 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.