Recursive Feature Elimination returns suspect results regarding accuracy

Question

I am doing RFE with 18 features. My understanding of RFE is, it starts by trying N - 1 models, and then removing the feature that causes the N - 1 models to perform worst. Then it tries N - 2 models, and so on. I've also read elsewhere that RFE does some sort of weighting. Which one is it? Weighting to me indicates a more math-y approach that requires more interpretation of the "optimal" features, whereas doing the N - 1, N - 2 indicates to me more of a brute force approach that requires less interpretation of the "optimal" features.

My model specifically said the highest accuracy was with 18 features. However, the suspect aspect was how the accuracy stayed the same when RFE was run with 0 through 15 features (see picture).

Does this indicate that something might be wrong with my RFE model? If so, what could be wrong?

Answer 1

That's not how RFE should work. Backwards RFE would build a model with all the variables and then iteratively remove 1 variable with the lowest effect. The plot is a little weird, however keep in mind that the difference between 25 and 18 features in the plot is only 0.003, easily just a random effect.