How to use 10-fold Cross Validation in Feature Selection?

Question

I would like to run a 10-fold cross validation on a number of different feature selection tools. For some tools, you can specify k-fold in the Python module (i.e., LassoLarsCV(cv=10)), but others it is not clear how to implement the cross-validation.

Let's assume, I divided my data into 10 random splits and run the feature selection in each fold. Doing so, there will be some set of variables (many same ones as well as new ones) in each fold. How do you cross-validate these nominal outcomes? They are not means or anything so we can take the average of the 10 fold, but all we have is a different number of variables as a result of validation in each fold. In other words, how can I validate the ideal set of variables in cross-validation procedure? Taking the features that are consistently found in each fold?

Answer 1

When you perform $k$-fold cross validation, you split the data equally and randomly into $k$ splits. Now you,

1. Take $i^{th}$ split as validation set, and combine the rest $k-1$ splits
2. Train on the $k-1$ splits combined, test on the validation set

Do this for $i = 1,..., k$ and note the average error. Repeat all these steps for each potential set of features, and then choose the set that gave you the lowest average error. Note that this requires you to go through $2^n$ combinations, where $n$ is the total number of features. If you can assume independence among the features, you can select them in a greedy fashion: you start with choosing just one feature. See which one among the $n$ features gives you the lowest error, and then keeping that constant, add one more from the remaining, do this $n-1$ times for the $n-1$ remaining features, and so on, until the error either doesn't decrease anymore, or the decrease is too low to offset the cost of increasing your feature space.

Answer 2

Let me clarify what cross-validation is.

In machine learning and predicting statistics in general, you propose a model to predict the data (which includes which features you use). To test your model (and your feature-selection), you run it on a dataset. To avoid a bias, you, of course, run it on unseen data and test its performance. Usually, you therefore split your data into a training and test set.

However, sometimes you do not have not enough data to be able to split it (you either lack training data then or the test set would be too small). That's where cross-validation (or k-folding) comes into play: you split your data into k folds, train on k-1 and predict on the last one. You repeat this k times. In the end, you get predictions for the whole dataset and it is "as if you used the whole dataset as test set". So your determine the performance by using all predictions for whatever metric you are using.

You can now do this (either train/test split if you have enough data or k-folding) with different sets of variables. The one with the best score is to be taken.

It is remarked here that you should usually have a validation set which you don't use in the whole training process: doing this kind of feature selection can bias your outcome! So having a validation set you did not use at all can give a good performance estimation in the end.