I think I have some sort of overfitting issue in my model and I cant work out what it could be. I creating a classification model for customer churn based on customer service variables. One of these variables is the length of time a customer has been with us. However, this variable is much more important than any other variable. This leads me to believe the model will assign current or churn based on this variable alone too much.
The train and test confusion matrix also looks too good to be true (1 = churn):
precision recall f1 Pred:0 0.9100418 0.9814982 0.9444203 Pred:1 0.9799315 0.9030221 0.9399061
I have also looked at the spread of number current and churned customers by groups of months the customers have been active but I cant really see a clear reason to churn based on this variable alone:
Status - Churn months active Count of Customers 0-9 724 10-19 1803 20-29 1725 30-39 1867 40-49 930 50-59 481 Status = Current Months active Count of Customers 0-9 4919 10-19 5418 20-29 4282 30-39 3664 40-49 25329 50-59 9354 110-119 1
Could anyone shed any light on a potential issue I have here or do you need more info?