I have a classification dataset, which I partitioned into a training and a test set. The test set I put away in a bank vault and have not touched since then, planning to use it only to evaluate the final model.
I do model selection and parameter tuning using 10-fold cross-validation on the training set. The question is, how should I perform feature selection? I am using Gradient Boosting Feature Selection, which is essentially a bunch of decision trees, where using features previously unused is penalized. This encourages the model to select a small subset of uncorrelated features.
The problem is, it can select different features during different rounds of cross-validation. For example, if I clone the same feature f several times as f1, f2, f3, during one round f may be selected, during another f1, etc. So averaging feature importances obtained with cross-validation would defeat the purpose of what I am trying to do.
Should I do feature selection on the whole training set, and then check if the features selected are "good enough" through cross-validation? But that can introduce bias, as the selected features may be overfit to my training set.
But if I just use one round of cross-validation to perform feature selection (and to test the selected features), the variance in assessing the performance of the model using the selected features could be very high.
So - what part of the data should I do feature selection on, and how do I verify that the selected features are good enough?