1
$\begingroup$

It is about Linear, Ridge and Lasso regressions.

If features are not standardized, then there is no way we can say something about the coefficients because the change in units (from cm to m) will either decrease or increase the coefficients.

So, it gets more interesting when features are standardized. It is stated in the article

.. when all features are on the same scale, the most important features should have the highest coefficients in the model, while features uncorrelated with the output variables should have coefficient values close to zero. [..] Ridge regression on the other hand can be used for data interpretation due to its stability and the fact that useful features tend to have non-zero coefficients.

However, the answer of @gung regarding the interpretation and stadatdization:

For instance, if $\beta_1=.6$, and $\beta_2=.3$, then the first explanatory variable is twice as important as the second. While this idea is appealing, unfortunately, it is not valid. There are several issues, but perhaps the easiest to follow is that you have no way to control for possible range restrictions in the variables.

Lets say, my features are standadized. In Linear, Ridge and Lasso I get $\beta_1=1.4$, and $\beta_2=.3$. Do the following statements hold for all three methods?

1.) A unit change in feature 1 has more effect on the response variable
2.) Feature 1 is more important than feature 2
3.) Feature 1 is more useful than feature 2

If in Lasso I get $\beta_1=0$, $\beta_2=.3$ and $\beta_3=1.4$. Would the following be right:

4.) Feature 1 is not important at all.
5.) Feature 3 is more important than feature 2?

$\endgroup$
  • 4
    $\begingroup$ "Important" here is an undefined term. If you define importance to mean the standardized coefficient is larger, than sure you can say that. Otherwise you need to have a clear definition of "importance" that does not depend on your model. I have found that most people fail at making the meaning of "important" clear. $\endgroup$ – Matthew Drury Jun 15 '17 at 23:36
  • $\begingroup$ @MatthewDrury I would define important variable as a variable that has more effect on the explained the response. That would mean actually that all above statements are true. Is it right? $\endgroup$ – Alina Jun 19 '17 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.