This question is a follow up for this one: Feature selection and cross-validation

If I do cross-validation of a process that includes model (features) selection (so I re-do model selection for each fold), can I use the results of this cross-validation to inform the final model selection on the whole training set?

For example, I may be more reluctant to include features that only were included in a few folds, or more lenient in including features that are marginal on the full model but were consistently included in most of the folds.

I have the feeling that this would be wrong, because cross validation evaluated the process, not the particular features of the model. Then again, I want to have a good model, so I feel inclined to use all the information available to select the final model.

Which approach is correct, if any? Would regularization be a better approach to feature selection? I thought about using bootstrapping as well, but I am using LOO cross-validation because I want/need to generate a prediction accuracy for each of my observations (which LOO does). Thanks.

Follow-up: I ended up ignoring the information gained from cross-validation. I chose not to include a "marginal" component based on better residuals distribution, ignoring the fact that it did get picked up quite regularly during cross-val. Not sure it was the right move, but I am happy with my model behavior. Besides, the process included PCA: the exact composition of each feature probably varied a bit for each fold, so it would have been risky or even wrong to use the information gained from cross-validation anyway.

  • $\begingroup$ Model selection and feature selection are very different things. Competing models may not necessarily be nested with the former, which is the case with AIC stepwise model selection or Bayesian Model Averaging, or so on. Feature selection from boosting, bagging, Lasso, has a much more regular space of nested models determined by inclusion or exclusion of specific variables. $\endgroup$ – AdamO Apr 24 '14 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.