I would like to run a 10-fold cross validation on a number of different feature selection tools. For some tools, you can specify k-fold in the Python module (i.e., LassoLarsCV(cv=10)), but others it is not clear how to implement the cross-validation.

Let's assume, I divided my data into 10 random splits and run the feature selection in each fold. Doing so, there will be some set of variables (many same ones as well as new ones) in each fold. How do you cross-validate these nominal outcomes? They are not means or anything so we can take the average of the 10 fold, but all we have is a different number of variables as a result of validation in each fold. In other words, how can I validate the ideal set of variables in cross-validation procedure? Taking the features that are consistently found in each fold?

  • 5
    $\begingroup$ You don't use CV to validate the actual features chosen, you use it to determine the optimal value of a hyperparameter (i.e. a regularization parameter, or the number of selected features). Once you know this, you fit a final model on all of your CV data together using the value of that parameter. $\endgroup$ – Matthew Drury May 12 '17 at 22:29

When you perform $k$-fold cross validation, you split the data equally and randomly into $k$ splits. Now you,

  1. Take $i^{th}$ split as validation set, and combine the rest $k-1$ splits
  2. Train on the $k-1$ splits combined, test on the validation set

Do this for $i = 1,..., k$ and note the average error. Repeat all these steps for each potential set of features, and then choose the set that gave you the lowest average error. Note that this requires you to go through $2^n$ combinations, where $n$ is the total number of features. If you can assume independence among the features, you can select them in a greedy fashion: you start with choosing just one feature. See which one among the $n$ features gives you the lowest error, and then keeping that constant, add one more from the remaining, do this $n-1$ times for the $n-1$ remaining features, and so on, until the error either doesn't decrease anymore, or the decrease is too low to offset the cost of increasing your feature space.

  • $\begingroup$ What is wrong about the answer? Do you mind going into a little more detail? $\endgroup$ – Antimony May 14 '17 at 6:48
  • $\begingroup$ I am unsure what you mean. Probably you got things right. Did you mean: "Then you do this for all possible combinations of features and the one that gave you the lowest validation error..."? If so, this is very unclear formulated, at least for me. $\endgroup$ – Mayou36 May 14 '17 at 19:08
  • $\begingroup$ Yes that is what I meant. Thanks for the feedback, I'll edit my answer. Next time say it isn't clear rather than saying that it is wrong. $\endgroup$ – Antimony May 14 '17 at 23:48
  • $\begingroup$ The way you formulated it made me assume it is wrong. Reading it a couple of times more made me think whether it is may right. But anyway, I removed my comment;) $\endgroup$ – Mayou36 May 16 '17 at 14:52

Let me clarify what cross-validation is.

In machine learning and predicting statistics in general, you propose a model to predict the data (which includes which features you use). To test your model (and your feature-selection), you run it on a dataset. To avoid a bias, you, of course, run it on unseen data and test its performance. Usually, you therefore split your data into a training and test set.

However, sometimes you do not have not enough data to be able to split it (you either lack training data then or the test set would be too small). That's where cross-validation (or k-folding) comes into play: you split your data into k folds, train on k-1 and predict on the last one. You repeat this k times. In the end, you get predictions for the whole dataset and it is "as if you used the whole dataset as test set". So your determine the performance by using all predictions for whatever metric you are using.

You can now do this (either train/test split if you have enough data or k-folding) with different sets of variables. The one with the best score is to be taken.

It is remarked here that you should usually have a validation set which you don't use in the whole training process: doing this kind of feature selection can bias your outcome! So having a validation set you did not use at all can give a good performance estimation in the end.

  • $\begingroup$ "The one with the best score is to be taken." Best score of what? All you have is a bunch of variable suggested by the feature selection algorithm you use, such as Boruta, ExtraTreeClassifier, etc.. $\endgroup$ – worthy May 15 '17 at 5:21
  • $\begingroup$ Hm, I don't think that ExtraTreeClassifier is a feature selection algorithm but a classification algorithm. There are two ways of tackling a problem: you can use a blackbox and read its description. Then you may specify what blackbox you are using. Or you understand what is going on and do it yourself. This is where my answer aims at. To be clear: the score refers to the metric you are using. There has to be something you want to optimise (ROC AUC, accuracy,...). But did you get the part about cross-validation? Cross-validation and feature selection are completely independant things... $\endgroup$ – Mayou36 May 16 '17 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.