2
$\begingroup$

In my test there was a false/true question:

Your estimated model for predicting house prices has a large positive weight on 'square feet living'. This implies that if we remove the feature 'square feet living' and refit the model, the new predictive performance will be worse than before.

In fact, if we change the units of this feature into sq meters, we could get a much lower positive weight. Meaning that the weight does not say anything about the importance of the feature.

However, in the ridge and lasso regression, the smaller the weight is, the less important is the feature. As far as I understand it, it implies only to ridge and lasso but not to a-non-regularized-regression (and therefore the answer is "false" to the question). Is it right?

$\endgroup$
  • 1
    $\begingroup$ (1) If you change from square feet to square meters, the coefficient will increase proportionately, not decrease. (2) Ridge and Lasso typically require you to standardize the regressors precisely to avoid this issue of arbitrariness due to choice of units of measurement. $\endgroup$ – whuber Jan 29 '17 at 23:04
3
$\begingroup$

In fact, if we change the units of this feature into sq meters, we could get a much lower positive weight. Meaning that the weight does not say anything about the importance of the feature.

You're right. The magnitude of the weight here is not invariant under change of units, and so cannot say much of consequence about anything.

However, in the ridge and lasso regression, the smaller the weight is, the less important is the feature.

In ridge and lasso we always standardize the features before running the algorithm, which makes them unitless. This is essential, else ridge and lasso fall prey to the same issues.

It's debatable whether, even in the standardized case, magnitudes of weights say much of anything about the importance of the feature, or whether removing or including the feature in the model will improve or decay out of sample performance. The only way to know the answer to that is to try it and see(*), you generally can't just look at the coefficients and tell.

(*) Except, don't try it too many times, or you'll over-fit to your holdout set.

$\endgroup$
  • $\begingroup$ Completely agree with your point that coefficient magnitudes don't indicate variable importance when predictors aren't standardized (+1). I do think there are cases where you wouldn't want to standardize (or would eventually convert back to original units). For example, without standardizing, a weight of $w$ for the 'square footage' variable would mean: holding all other variables constant, you'd expect price to increase by $w$ dollars if area increases by $w$ square feet. I could imagine that being useful for certain problems, and undesirable for others. $\endgroup$ – user20160 Jan 30 '17 at 6:19
1
$\begingroup$

Multicollinearity is another issue to consider. If predictors are linearly dependent (i.e. perfect multicollinearity), ordinary least squares has no unique solution; it's possible for completely different choices of coefficients to be equally valid (and there are infinitely many such choices). If the predictors are highly correlated (i.e. multicollinear, but not perfectly so), the solution can still be unstable. In these cases, it doesn't make sense to draw conclusions based on the magnitudes of OLS coefficients. This is true even if the predictors are standardized.

When two predictors are highly correlated, lasso may include one in the model but not the other. Therefore, if multicollinearity is present, a lasso weight of zero doesn't imply that a predictor is unimportant; it could instead be correlated with another selected predictor.

In contrast, ridge regression will distribute the weight among correlated features. But, care must still be taken when interpreting weights if multicollinearity is present. For example, consider adding multiple identical copies of a predictor to a data set. The identical predictors will receive identical weights. But, these weights will have smaller magnitude than the case where only one copy is present.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.