I' want to train a model for classifiation. I have a pretty small data-set (about 400 data unevenly distributed among three classes). To evaluate my model, i want to perform cross-validation, with a step of feature-selection (by wrapper) for each fold. I read that using different data for feature selection and for training the model (and of course for validation) is better in order to avoid overfiting.

Do you agree, or spliting the data in two set, one for feature selection and training, and the other for validation is good enough ?

If not, what proportion of data should i use for the feature selection - training - validation set of each fold ?




Why do you want feature selection?

If you are building a predictive model, even if you are not using a feature selection method, it is highly advised to have training(2/3 of your whole data) and validation sets(1/3 of your whole data for example).

Feature selection, in my opinion is very risky. Especially if the selection takes place by looking at the responses which are classes in your case. So selecting a set of features which corresponds to the lowest classification error according to a cross-validation will mostly likely end up with over-fitting.

Assume this is not the case, then, you would build a model with these variables and validate this model with your independent validation set. The problem here is, if the results are not good, people adjust parameters and apply another algorithm for feature selection until they get low classification errors on "independent" validation set which actually can not be called "independent" anymore... Finally you may end with a model that overfits both to your training and to your test sets.

You may want to use methods like Lasso and ElasticNet which does the feature selection in a more robust and in overfitting-wise safer way.

  • $\begingroup$ Ok, got it. I want to do feature selection because i have many redundant variables, and i want to know which ones fit better with the classification problem. $\endgroup$ – mprl Jun 2 '17 at 12:15
  • 1
    $\begingroup$ You can for example use PLS-DA and check the coefficients to evaluate which ones are more important. But if your aim is to continue with a lower number of variables (which have no guarantee to yield a better model) than Lasso is way to go in my opinion. $\endgroup$ – theGD Jun 2 '17 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.