In a recent conversation with one of the colleagues I was presented with a view that LASSO/Ridge regularization (trading bias for variance) renders coefficient estimates useless for interpretation, i.e. LASSO/Ridge coefficients can't be interpreted in the same way as OLS coefficients due to increase in bias. While I don't agree with this, I'd like to hear some thoughts on this to prepare a more coherent counterargument.
$\begingroup$
$\endgroup$
4
asked Mar 23 at 0:19
-
$\begingroup$ The edited title strikes me as better reflecting the question, since the bias-variance decomposition is rather unrelated, though it is easy to edit it back if you do not like it. $\endgroup$– DaveMar 23 at 0:38
-
1$\begingroup$ What does this colleague mean by "interpreting" OLS coefficients? $\endgroup$– FirebugMar 23 at 10:09
-
$\begingroup$ Interpreting in the standard fashion: a unit change in X leads to a change in Y equal to coefficient for X. I guess the mindset here is that due to introduction of bias coefficients somehow lose "physical" meaning associated with underlying real world processes. $\endgroup$– Always Right Never LeftMar 23 at 10:20
-
$\begingroup$ Not at all! The interpretation of a coefficient does not depend on how that coefficient is estimated -- the coefficient still means the same thing it always did. $\endgroup$– whuber ♦Mar 23 at 13:31
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
I totally disagree with this.
Many common estimators are biased. Ever calculated $S = \sqrt{\dfrac{\overset{n}{\underset{i=1}{\sum}}\left(X_i-\bar X\right)^2}{n-1}}$ as the standard deviation? That is a biased estimator for the standard deviation. Ever fit a logistic regression? You used a biased estimator of the true coefficients. Despite this, however, you probably felt comfortable interpreting those statistics (as is reasonable). Consider asking your colleague about these. (I concede that the Workplace Stack Exchange might advise otherwise.) If your colleague protests, "But they are consistent," so are ridge and LASSO. (Yes, that requires some assumptions, but OLS unbiasedness requires assumptions, too.)
This is subtle, but I dispute the idea that you are interpreting the estimator. You interpret the model parameters, and you use estimators to guess what those unknown parameters are. The ridge and LASSO estimators are just estimating the $\beta$ parameter vector of $\mathbb E\left[Y\vert X\right] = X\beta$, same as OLS. It might be that you are in a situation where ridge or LASSO has inferior properties compared to the OLS estimator (maybe a reviewer insists on an unbiased estimator), but all of these are just ways of guessing the true values of the parameters.
If you have a reason to estimate a certain way (ridge, LASSO, OLS, something else), I say that you estimate that way and interpret using your guess of the parameter(s).
-
3$\begingroup$ "But they are consistent," so are ridge and LASSO. Well, perhaps they are, perhaps they are not... LASSO and ridge coefficients are only consistent under a certain subset of rules for selecting regularization intensity. Actually, they may be biased even for standard rules such as optimizing prediction performance using cross validation; see Asymptotic bias of LASSO vs. none of SCAD. $\endgroup$ Mar 23 at 9:11