Recursive Feature Elimination in sklearn

Question

I have been thinking about one thing after reading documentation from sklearn about Feature Selection for building prediction models (http://scikit-learn.org/stable/modules/feature_selection.html#rfe)

More specifically 1.13.3 --> Recursive Feature Elimination

The criterion used to eliminate variables is the coef_, which is the value of the coefficients for the parameters of your model. If you loop this process, each time removing the last in the ranking that the method returns, you are supposed to be keeping the most relevant parameters for your model.

My question now is: what does the value of the coefficient have to do with its importance? It is just a number that is going to be multiplied by the actual value of the variable. So, variables like for example temperature in Kelvins can have values in some processes of 1200K (just as an example), while the pressure in the same process is maybe 5 bar. So it is quite easy that the coefficient that goes with temperature is going to be lower, so after being multiplied by 1200 it has a reasonable value to predict, let's say, some force in Newtons, but the pressure coefficient is going to have a smaller value. This does not mean the temperature is less relevant for the response than the pressure!

So, where is the fundament of the FSE then?

Answer 1

You are quite correct. Using raw coefficients doesn't make sense, because these are not comparable.

Unless, that is, the predictors have been normalized - in this case, you have standardized coefficients, which are comparable.

Alternatively, the page you link to suggests using the feature_importances_ attribute. Variable importance makes a lot more sense. (It often also takes a long time to calculate, if you use permutations and out-of-bag accuracy, so if you want to use this in a recursive way, performance may be a bottleneck.)