I am running a customer churn predictive model in r. My confusion is when I try different combinations of variables I.e. Removing some from the model, I get completely different results in terms of roc,Auc and precision etc.

Do I just select the variables that give me the best results in terms of Auc,roc? My issue with this is if I include all 10 variables, some of them are much more important than the others and may be causing over fitting as the train and test results are very high.

How can I validate some of this?

  • 1
    $\begingroup$ I'm an economist and I would usually say that you should include what makes sense to you. If you worry about overfitting, why not reduce your sample size, estimate the model and then compare performance for the out of sample prediction? $\endgroup$
    – Felix H
    Commented Nov 28, 2016 at 8:57
  • 1
    $\begingroup$ Are you trying to kill performance? Variable/feature selection will help you on that. $\endgroup$
    – utobi
    Commented Nov 28, 2016 at 11:46
  • $\begingroup$ @utobi: could you elaborate on your point? $\endgroup$
    – Metariat
    Commented Nov 28, 2016 at 12:25
  • $\begingroup$ Assuming that all the features you are given are potentially useful for predicting the response in question, in my opinion, the only practical justification of feature screening is when you have a super high number of features that you can't process otherwise. In all other cases, dropping features using whatever machine learning or statistical criteria is not worth the effort. Double cross-validating for model tuning and then for feature selection will make you go into multiplicity issues and the gain, if there is such a gain, in terms of predictive performance, will be biased. $\endgroup$
    – utobi
    Commented Nov 28, 2016 at 13:22
  • $\begingroup$ @utobi: does it mean that we should include all variables in the model, even if it's, lets say, the name of the sister in the family, which has nothing to do with the churn? $\endgroup$
    – Metariat
    Commented Nov 28, 2016 at 17:17

3 Answers 3


I would suggest the following approach:

  1. First of all, it is important to define what is your criterion. Is it accuracy? Is it to maximize recall and to assure that the precision is at least 80%? This decision may be dependent on your context. For example, if you want to detect cancer, you want to focus on recall because it is dangerous to omit a potential patient.
  2. Divide the data you have into training, validation, and test sets. If you data set is sufficiently large, you do not have to go for cross-validation.
  3. For each possible combination of variables, you can train the model on training data, tune with validation data, and finally compare on test data. Then, you know which variables are more relevant.

Some important notes:

  • The feature selection can be understood also as a hyper parameter optimization. Thus, you can optimize not just the selection of features, but also e.g. number of neurons in neural network or number of trees in random forests.
  • One efficient way how to select the features is regularization that works well not just for logistic regression, but also for neural networks. The lasso can help.
  • Without splitting into training a test sets, the results can be confusing and blind trying of different subsets shall be avoided.
  • Sometimes, it may be helpful to see what features have a direct or indirect influence on the prediction. For this purpose, (i) Bayesian networks or (ii) careful treatment of multicolinearity can be considered.

A relevant reading is also here: http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf


The question is, "How to select predictor variables for a classification model?"

In order to give a specific answer, more information would eventually be needed about your dataset and specific application.

In churn analysis, the goal for a predictive model is achieve the highest prediction score possible. Variable selection helps to create simpler models, but can increase or decrease performance.

Variable selection occurs in multiple stages:

  • Variable selection in the data pre-processing stage
  • Variable selection in the model building stage

"Variable selection" includes the process of removing variables that fail to meet the assumptions of your model (i.e. non-normality, imbalanced classes, multicollinearity) and you may have to remove some variables or gather more data.

Potential Problems

  • Multicollinearity can lead to inflated variable importance
  • Too many variables for too few samples can be overfitting
  • Imbalanced classes can lead to poor performance
  • etc...

If you have problems such as these in your data, then variable selection may produce drastically differing results each time you train the model.

Once all your model assumptions are met, here are some techniques that may help. Keep in mind that each has its own benefits and drawbacks depending on your particular application and prediction model.

More variables

  • Try adding new variables using data from outside sources (i.e. weather, city data)
  • Try variable transformations
  • Try creating new features from existing data
  • Try higher order prediction models

There are many other approaches to this problem, hopefully this will give insight to some people :)


Irrespective of how you choose your variables, you should perform cross-validation on your resultant model to ensure it performs as well on data it wasn't built on.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.