What proportion of data for feature selection

Question

I' want to train a model for classifiation. I have a pretty small data-set (about 400 data unevenly distributed among three classes). To evaluate my model, i want to perform cross-validation, with a step of feature-selection (by wrapper) for each fold. I read that using different data for feature selection and for training the model (and of course for validation) is better in order to avoid overfiting.

Do you agree, or spliting the data in two set, one for feature selection and training, and the other for validation is good enough ?

If not, what proportion of data should i use for the feature selection - training - validation set of each fold ?

Thanks,

Maxime

Answer 1

Why do you want feature selection?

If you are building a predictive model, even if you are not using a feature selection method, it is highly advised to have training(2/3 of your whole data) and validation sets(1/3 of your whole data for example).

Feature selection, in my opinion is very risky. Especially if the selection takes place by looking at the responses which are classes in your case. So selecting a set of features which corresponds to the lowest classification error according to a cross-validation will mostly likely end up with over-fitting.

Assume this is not the case, then, you would build a model with these variables and validate this model with your independent validation set. The problem here is, if the results are not good, people adjust parameters and apply another algorithm for feature selection until they get low classification errors on "independent" validation set which actually can not be called "independent" anymore... Finally you may end with a model that overfits both to your training and to your test sets.

You may want to use methods like Lasso and ElasticNet which does the feature selection in a more robust and in overfitting-wise safer way.