derive importance of feature by its coefficient (multiple linear regression)

Question

In my test there was a false/true question:

Your estimated model for predicting house prices has a large positive weight on 'square feet living'. This implies that if we remove the feature 'square feet living' and refit the model, the new predictive performance will be worse than before.

In fact, if we change the units of this feature into sq meters, we could get a much lower positive weight. Meaning that the weight does not say anything about the importance of the feature.

However, in the ridge and lasso regression, the smaller the weight is, the less important is the feature. As far as I understand it, it implies only to ridge and lasso but not to a-non-regularized-regression (and therefore the answer is "false" to the question). Is it right?

Answer 1

In fact, if we change the units of this feature into sq meters, we could get a much lower positive weight. Meaning that the weight does not say anything about the importance of the feature.

You're right. The magnitude of the weight here is not invariant under change of units, and so cannot say much of consequence about anything.

However, in the ridge and lasso regression, the smaller the weight is, the less important is the feature.

In ridge and lasso we always standardize the features before running the algorithm, which makes them unitless. This is essential, else ridge and lasso fall prey to the same issues.

It's debatable whether, even in the standardized case, magnitudes of weights say much of anything about the importance of the feature, or whether removing or including the feature in the model will improve or decay out of sample performance. The only way to know the answer to that is to try it and see(*), you generally can't just look at the coefficients and tell.

(*) Except, don't try it too many times, or you'll over-fit to your holdout set.

Answer 2

Multicollinearity is another issue to consider. If predictors are linearly dependent (i.e. perfect multicollinearity), ordinary least squares has no unique solution; it's possible for completely different choices of coefficients to be equally valid (and there are infinitely many such choices). If the predictors are highly correlated (i.e. multicollinear, but not perfectly so), the solution can still be unstable. In these cases, it doesn't make sense to draw conclusions based on the magnitudes of OLS coefficients. This is true even if the predictors are standardized.

When two predictors are highly correlated, lasso may include one in the model but not the other. Therefore, if multicollinearity is present, a lasso weight of zero doesn't imply that a predictor is unimportant; it could instead be correlated with another selected predictor.

In contrast, ridge regression will distribute the weight among correlated features. But, care must still be taken when interpreting weights if multicollinearity is present. For example, consider adding multiple identical copies of a predictor to a data set. The identical predictors will receive identical weights. But, these weights will have smaller magnitude than the case where only one copy is present.