# Feature selection: which portion of the dataset to use?

I have a classification dataset, which I partitioned into a training and a test set. The test set I put away in a bank vault and have not touched since then, planning to use it only to evaluate the final model.

I do model selection and parameter tuning using 10-fold cross-validation on the training set. The question is, how should I perform feature selection? I am using Gradient Boosting Feature Selection, which is essentially a bunch of decision trees, where using features previously unused is penalized. This encourages the model to select a small subset of uncorrelated features.

The problem is, it can select different features during different rounds of cross-validation. For example, if I clone the same feature f several times as f1, f2, f3, during one round f may be selected, during another f1, etc. So averaging feature importances obtained with cross-validation would defeat the purpose of what I am trying to do.

Should I do feature selection on the whole training set, and then check if the features selected are "good enough" through cross-validation? But that can introduce bias, as the selected features may be overfit to my training set.

But if I just use one round of cross-validation to perform feature selection (and to test the selected features), the variance in assessing the performance of the model using the selected features could be very high.

So - what part of the data should I do feature selection on, and how do I verify that the selected features are good enough?

• Take a look e.g. here. Use your 10-fold CV on the training set, as you are describing, to fix the parameters of the GBFS: max depth of the trees, learning rate, trade-off parameter and n. of iterations. Then with those parameters, take the whole training set and perform GBFS; those will be the features you select. Cheers. Apr 19, 2017 at 12:43

First, lets denote your initial training set as $X$. What i would recommend is to split $X$ into $X_1$ (training set) and $X_2$ (development set), where $X_1$ should take $\approx$ 70% of the total size of $X$.
Use $X_1$ to train your model and $X_2$ to select features and/or learn other/additional hyperparameters. Once, you learned anything relevant for your model, you can apply it on the evaluation set $Z$. Never ever train on $Z$, don't even look into its underlying data.
When you observed good performance on $Z$ (e.g., through 10-fold cross-validation, leave-one-out, bootstrapping, or what ever) after the feature selection on $X_2$, it might reflect that its generalization ability worked and that your selected features were "good enough"$\ldots$
• By $Z$ you mean the test set, that is outside of $X$? Or you mean that I should separate my initial dataset into training $X$, validation $Z$ and test set $T$? But why do I need a validation set, could I not just do cross-validation on $X$ - isn't that the idea behind cross-validation anyway? Apr 19, 2017 at 9:23