This question is a follow up for this one: Feature selection and cross-validation
If I do cross-validation of a process that includes model (features) selection (so I re-do model selection for each fold), can I use the results of this cross-validation to inform the final model selection on the whole training set?
For example, I may be more reluctant to include features that only were included in a few folds, or more lenient in including features that are marginal on the full model but were consistently included in most of the folds.
I have the feeling that this would be wrong, because cross validation evaluated the process, not the particular features of the model. Then again, I want to have a good model, so I feel inclined to use all the information available to select the final model.
Which approach is correct, if any? Would regularization be a better approach to feature selection? I thought about using bootstrapping as well, but I am using LOO cross-validation because I want/need to generate a prediction accuracy for each of my observations (which LOO does). Thanks.
Follow-up: I ended up ignoring the information gained from cross-validation. I chose not to include a "marginal" component based on better residuals distribution, ignoring the fact that it did get picked up quite regularly during cross-val. Not sure it was the right move, but I am happy with my model behavior. Besides, the process included PCA: the exact composition of each feature probably varied a bit for each fold, so it would have been risky or even wrong to use the information gained from cross-validation anyway.