I want to understand the algorithm of recursive feature eliminiation (RFE) combined with crossvalidation (CV). An original source by Guyon et al. on RFE can be found here.

  • My understanding of RFE: We train our classifier - say a linear Support Vector Machine - first with all features. This gives us a weight for each feature. The absolute value of these weights reflects the importance of each feature. We remove the least important feature, perform again a training, get a new ranking and continue so on until we have ranked all our features

  • My question: I am running RFE cross-validated (in python with this implementation). In the example below, several features are ranked first. How does this come about? For a final ranking I assume that the RFE elimination has to be done repeatedly, so does this imply several applications of RFE where each time another feature has been ranked first? How is this combined with cross-validation, and how then is the (see plot below) classification accuracy from 1,2,3,..features calculated, when each of these subsets could consist of different features?

enter image description here

  • $\begingroup$ Your cross-validation is for tuning the model and seems to suggest that $3$ features may be a good choice. If several of the features are highly correlated, then it is not a surprise that most of them are eliminated or even that different ones remain for the various cross-validation folds. You do not need to worry about this for your final training step, so long as you remember that this is a prediction exercise, not one aimed at describing causality $\endgroup$
    – Henry
    Sep 27, 2017 at 9:30

1 Answer 1


Say you run a 3-fold RFECV. For each split, the train set will be transformed by RFE n times (for each possible 1..n number of features). The classifier supplied will be trained on the training set, and a score will be computed on the test set. Eventually, for each 1..n number of features, the mean result from the 3 different splits is shown on the graph you included. Then, RFEVC transforms the entire set using the best scoring number of features. The ranking you see is based on that final transformation.

  • $\begingroup$ Thanks for the quick answer! So say we have 4 features, and as you suggest 3-fold RFECV. Looking at the first fold: Do we train first with all 4, then with 3, then with 2, then with 1 features on this same training set, and subsequently for each of these test the classifier 4 times on the same test set (and then move on to the next fold)? $\endgroup$
    – Pugl
    Sep 27, 2017 at 11:42
  • $\begingroup$ @Pegah exactly. $\endgroup$
    – Eran
    Sep 27, 2017 at 11:53
  • 1
    $\begingroup$ @Pegah Actually, since each fold only uses the number of features, and not the features themselves, overfitting is avoided. If each fold produces the highest score with 3 features (without pre-specifying which features to use), it is reasonable to assume that using 3 features is a good choice. $\endgroup$
    – Eran
    Sep 27, 2017 at 13:45
  • 1
    $\begingroup$ Hi Eran, a bit late but I have one further question:) Your explanation is crystal clear, but one point I am still lacking in my understanding: Say this mean you mention results in having x in N as the optimal number features - how is then determined however which x features? As we know, the top features can vary from split to split.. $\endgroup$
    – Pugl
    May 2, 2018 at 16:02
  • 3
    $\begingroup$ Hi @Pegah :) RFECV only uses CV to select the optimal number of features to select. It then uses regular RFE (on the entire set) to actually preform selection. $\endgroup$
    – Eran
    May 3, 2018 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.