5
$\begingroup$

L1 penalized regression introduces a bias on your regression model but decreases the variance. When this bias is introduced, is it possible that the coefficient of $B$ changes sign? This would greatly affect the interpretation, where a regular regression might tell you an increase in $x$ increases $y$ by $B$, but maybe using lasso an increase in $x$ decreases $y$ by $B$

$\endgroup$
  • 3
    $\begingroup$ I don't think so; if you compute the LASSO path from the least squares estimates to some point with high shrinkage, if a coefficient is shrunk so much that it would switch sign, at the point at which the sign change takes place, it is removed from the model (i.e. fixed at 0). This is in contrast to ridge regression where shrinkage can switch signs IIRC. $\endgroup$ – Reinstate Monica - G. Simpson Jan 18 '13 at 20:26
  • 1
    $\begingroup$ @Gavin That sounds good, but what if the coefficient that is removed from the model is for an interaction? It seems possible that removing it could then affect the signs of the underlying variables in almost any conceivable way. $\endgroup$ – whuber Jan 18 '13 at 21:47
  • 1
    $\begingroup$ @whuber I guess that is just one of the problems with automated methods. I note that there are now grouped lasso methods and related techniques that work on groups of predictors, possibly in a hierarchy, that appear to attack this problem. $\endgroup$ – Reinstate Monica - G. Simpson Jan 18 '13 at 22:11
  • $\begingroup$ Isn't it important to consider also the size of the effect? If the coefficient is very close to 0 (on a scale that makes substantive sense) then do we care if the sign is positive or negative? $\endgroup$ – Peter Flom - Reinstate Monica Jan 19 '13 at 15:45
  • 1
    $\begingroup$ @PiotrSokol I just meant that the scale on which the IV was measured was sensible. E.g., for "weight of adult humans" pounds or kg makes sense. Tons does not make sense. In addition, some scales are purely arbitrary. $\endgroup$ – Peter Flom - Reinstate Monica Apr 24 '13 at 22:55
6
$\begingroup$

In the original paper of Tibshirani (1995) it says that the signs of the LASSO estimate and the least squares solution may be different (p270 last paragraph).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.