What impact does a much higher importance for one variable have on a predictive model?

I think I have some sort of overfitting issue in my model and I cant work out what it could be. I creating a classification model for customer churn based on customer service variables. One of these variables is the length of time a customer has been with us. However, this variable is much more important than any other variable. This leads me to believe the model will assign current or churn based on this variable alone too much.

The train and test confusion matrix also looks too good to be true (1 = churn):

      precision recall    f1
Pred:0 0.9100418 0.9814982 0.9444203
Pred:1 0.9799315 0.9030221 0.9399061


I have also looked at the spread of number current and churned customers by groups of months the customers have been active but I cant really see a clear reason to churn based on this variable alone:

Status - Churn

months active   Count of Customers
0-9                  724
10-19               1803
20-29               1725
30-39               1867
40-49               930
50-59                481

Status  = Current

Months active   Count of Customers
0-9                  4919
10-19                5418
20-29                4282
30-39                3664
40-49                25329
50-59                9354
110-119               1


Could anyone shed any light on a potential issue I have here or do you need more info?

Many thanks.

• Have you tried to get results excluding this variable? Nov 18 '16 at 7:50
• Hi, yes. The importance of the variables are much more balanced without the month active variable and the precision/recall etc reduces to a more realistic number - however, how do i know if this is the right thing to do? Maybe the months active really is that important and needs to stay in? Just not sure how to measure it. Thanks Nov 18 '16 at 8:04

Short answer: Your confusion matrix is OK. The issue you are having is the nature of your data. You need to investigate it further.

1. Express the churners as a percentage of total customers in every bin/group (Months Active)
|         | Non-churn | Churn |     |
|---------|-----------|-------|-----|
| 0-9     | 4919      | 724   | 15% |
| 10-19   | 5418      | 1803  | 33% |
| 20-29   | 4282      | 1725  | 40% |
| 30-39   | 3664      | 1867  | 51% |
| 40-49   | 25329     | 930   | 4%  |
| 50-59   | 9354      | 481   | 5%  |
| 110-119 | 1         |       | 0%  |


Now, the first question you need to ask yourself is why they are so unevenly distributed. Both churner and non-churner numbers look strange. You need to understand why you have 25329 customers in 40~49 and 3664 in 30~39? Maybe you can find the explanation in the way your data was gathered.

1. Your issue comes from the fact that you capture the churners differently in the different bins/groups. E.g. If I take bin 50-59 (5% churners) and 30-39 (51% churners), I would say that a customer who is active 35 months, is much more likely to churn than a customer who is active 54 months. - This is exactly what your model tells you.

Therefore, you need to consider if your bins/groups are suitable for your data. (Maybe you should group them by a different column. - This is just an idea what you can look for.)