I want to understand the algorithm of recursive feature eliminiation (RFE) combined with crossvalidation (CV). An original source by Guyon et al. on RFE can be found here.

  • My understanding of RFE: We train our classifier - say a linear Support Vector Machine - first with all features. This gives us a weight for each feature. The absolute value of these weights reflects the importance of each feature. We remove the least important feature, perform again a training, get a new ranking and continue so on until we have ranked all our features

  • My question: I am running RFE cross-validated (in python with this implementation). In the example below, several features are ranked first. How does this come about? For a final ranking I assume that the RFE elimination has to be done repeatedly, so does this imply several applications of RFE where each time another feature has been ranked first? How is this combined with cross-validation, and how then is the (see plot below) classification accuracy from 1,2,3,..features calculated, when each of these subsets could consist of different features?

enter image description here

  • $\begingroup$ Your cross-validation is for tuning the model and seems to suggest that $3$ features may be a good choice. If several of the features are highly correlated, then it is not a surprise that most of them are eliminated or even that different ones remain for the various cross-validation folds. You do not need to worry about this for your final training step, so long as you remember that this is a prediction exercise, not one aimed at describing causality $\endgroup$ – Henry Sep 27 '17 at 9:30

Say you run a 3-fold RFECV. For each split, the train set will be transformed by RFE n times (for each possible 1..n number of features). The classifier supplied will be trained on the training set, and a score will be computed on the test set. Eventually, for each 1..n number of features, the mean result from the 3 different splits is shown on the graph you included. Then, RFEVC transforms the entire set using the best scoring number of features. The ranking you see is based on that final transformation.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the quick answer! So say we have 4 features, and as you suggest 3-fold RFECV. Looking at the first fold: Do we train first with all 4, then with 3, then with 2, then with 1 features on this same training set, and subsequently for each of these test the classifier 4 times on the same test set (and then move on to the next fold)? $\endgroup$ – Pugl Sep 27 '17 at 11:42
  • $\begingroup$ @Pegah exactly. $\endgroup$ – Eran Sep 27 '17 at 11:53
  • $\begingroup$ Last question:) Aren't we then kind of overfitting when we re-use the test/training sets in this way? $\endgroup$ – Pugl Sep 27 '17 at 13:10
  • 1
    $\begingroup$ @Pegah Actually, since each fold only uses the number of features, and not the features themselves, overfitting is avoided. If each fold produces the highest score with 3 features (without pre-specifying which features to use), it is reasonable to assume that using 3 features is a good choice. $\endgroup$ – Eran Sep 27 '17 at 13:45
  • 2
    $\begingroup$ Hi @Pegah :) RFECV only uses CV to select the optimal number of features to select. It then uses regular RFE (on the entire set) to actually preform selection. $\endgroup$ – Eran May 3 '18 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.