# Understanding cross-validated recursive feature elimination

I want to understand the algorithm of recursive feature eliminiation (RFE) combined with crossvalidation (CV). An original source by Guyon et al. on RFE can be found here.

• My understanding of RFE: We train our classifier - say a linear Support Vector Machine - first with all features. This gives us a weight for each feature. The absolute value of these weights reflects the importance of each feature. We remove the least important feature, perform again a training, get a new ranking and continue so on until we have ranked all our features

• My question: I am running RFE cross-validated (in python with this implementation). In the example below, several features are ranked first. How does this come about? For a final ranking I assume that the RFE elimination has to be done repeatedly, so does this imply several applications of RFE where each time another feature has been ranked first? How is this combined with cross-validation, and how then is the (see plot below) classification accuracy from 1,2,3,..features calculated, when each of these subsets could consist of different features?

• Your cross-validation is for tuning the model and seems to suggest that $3$ features may be a good choice. If several of the features are highly correlated, then it is not a surprise that most of them are eliminated or even that different ones remain for the various cross-validation folds. You do not need to worry about this for your final training step, so long as you remember that this is a prediction exercise, not one aimed at describing causality – Henry Sep 27 '17 at 9:30