How to select predictor variables for a classification model? I am running a customer churn predictive model in r. My confusion is when I try different combinations of variables I.e. Removing some from the model, I get completely different results in terms of roc,Auc and precision etc. 
Do I just select the variables that give me the best results in terms of Auc,roc? My issue with this is if I include all 10 variables, some of them are much more important than the others and may be causing over fitting as the train and test results are very high.
How can I validate some of this? 
 A: I would suggest the following approach:


*

*First of all, it is important to define what is your criterion. Is it accuracy? Is it to maximize recall and to assure that the precision is at least 80%? This decision may be dependent on your context. For example, if you want to detect cancer, you want to focus on recall because it is dangerous to omit a potential patient. 

*Divide the data you have into training, validation, and test sets. If you data set is sufficiently large, you do not have to go for cross-validation.

*For each possible combination of variables, you can train the model on training data, tune with validation data, and finally compare on test data. Then, you know which variables are more relevant.


Some important notes:


*

*The feature selection can be understood also as a hyper parameter optimization. Thus, you can optimize not just the selection of features, but also e.g. number of neurons in neural network or number of trees in random forests.

*One efficient way how to select the features is regularization that works well not just for logistic regression, but also for neural networks. The lasso can help.

*Without splitting into training a test sets, the results can be confusing and blind trying of different subsets shall be avoided.

*Sometimes, it may be helpful to see what features have a direct or indirect influence on the prediction. For this purpose, (i) Bayesian networks or (ii) careful treatment of multicolinearity can be considered.


A relevant reading is also here:
http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf
A: 
The question is, "How to select predictor variables for a classification model?"

In order to give a specific answer, more information would eventually be needed about your dataset and specific application.
In churn analysis, the goal for a predictive model is achieve the highest prediction score possible. Variable selection helps to create simpler models, but can increase or decrease performance.

Variable selection occurs in multiple stages:

*

*Variable selection in the data pre-processing stage

*Variable selection in the model building stage


"Variable selection" includes the process of removing variables that fail to meet the assumptions of your model (i.e. non-normality, imbalanced classes, multicollinearity) and you may have to remove some variables or gather more data.

Potential Problems

*

*Multicollinearity can lead to inflated variable importance

*Too many variables for too few samples can be overfitting

*Imbalanced classes can lead to poor performance

*etc...

If you have problems such as these in your data, then variable selection may produce drastically differing results each time you train the model.

Once all your model assumptions are met, here are some techniques that may help.  Keep in mind that each has its own benefits and drawbacks depending on your particular application and prediction model.

More variables

*

*Try adding new variables using data from outside sources (i.e. weather, city data)

*Try variable transformations

*Try creating new features from existing data

*Try higher order prediction models


There are many other approaches to this problem, hopefully this will give insight to some people :)
A: Irrespective of how you choose your variables, you should perform cross-validation on your resultant model to ensure it performs as well on data it wasn't built on.
