I have a classification dataset, which I partitioned into a training and a test set. The test set I put away in a bank vault and have not touched since then, planning to use it only to evaluate the final model.

I do model selection and parameter tuning using 10-fold cross-validation on the training set. The question is, how should I perform feature selection? I am using Gradient Boosting Feature Selection, which is essentially a bunch of decision trees, where using features previously unused is penalized. This encourages the model to select a small subset of uncorrelated features.

The problem is, it can select different features during different rounds of cross-validation. For example, if I clone the same feature f several times as f1, f2, f3, during one round f may be selected, during another f1, etc. So averaging feature importances obtained with cross-validation would defeat the purpose of what I am trying to do.

Should I do feature selection on the whole training set, and then check if the features selected are "good enough" through cross-validation? But that can introduce bias, as the selected features may be overfit to my training set.

But if I just use one round of cross-validation to perform feature selection (and to test the selected features), the variance in assessing the performance of the model using the selected features could be very high.

So - what part of the data should I do feature selection on, and how do I verify that the selected features are good enough?

  • $\begingroup$ Take a look e.g. here. Use your 10-fold CV on the training set, as you are describing, to fix the parameters of the GBFS: max depth of the trees, learning rate, trade-off parameter and n. of iterations. Then with those parameters, take the whole training set and perform GBFS; those will be the features you select. Cheers. $\endgroup$
    – lrnzcig
    Commented Apr 19, 2017 at 12:43

1 Answer 1


First, lets denote your initial training set as $X$. What i would recommend is to split $X$ into $X_1$ (training set) and $X_2$ (development set), where $X_1$ should take $\approx$ 70% of the total size of $X$.

Use $X_1$ to train your model and $X_2$ to select features and/or learn other/additional hyperparameters. Once, you learned anything relevant for your model, you can apply it on the evaluation set $Z$. Never ever train on $Z$, don't even look into its underlying data.

When you observed good performance on $Z$ (e.g., through 10-fold cross-validation, leave-one-out, bootstrapping, or what ever) after the feature selection on $X_2$, it might reflect that its generalization ability worked and that your selected features were "good enough"$\ldots$

  • $\begingroup$ By $Z$ you mean the test set, that is outside of $X$? Or you mean that I should separate my initial dataset into training $X$, validation $Z$ and test set $T$? But why do I need a validation set, could I not just do cross-validation on $X$ - isn't that the idea behind cross-validation anyway? $\endgroup$
    – rinspy
    Commented Apr 19, 2017 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.