2
$\begingroup$

I can understand that lasso could make some weights to zero and prevent over-fitting. But for all the figure I see about lasso regression, weight will stay at zero once it reach zero and increasing lambda won't drive it to negative? Why is that?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

Let's say that $\beta$ is the vector of coefficients to be estimated by Lasso regression.

Lasso regression applies an $L1$ penalty to the coefficients. The penalty term in the objective function is $\lambda\|\beta\|_1 = \lambda \Sigma |\beta_i|$. I.e., the absolute values of the coefficients are being penalized. Therefore, negative coefficients incur the same penalty as positive coefficients of this same magnitude. So it is the absolute values of the coefficients which are driven down toward zero.

Note that an alternative form for Lasso regression is to apply the penalty term in the form of a constraint $\Sigma |\beta_i| \le t$, for some $t > 0$, rather than as an additive term in the objective function. As with the objective function form of penalty, it is the absolute value of the estimated coefficients which are being constrained (in effect, penalized). Note that for any value of $\lambda$ in the objective function penalty term, there would be some corresponding value of $t$ in the "constraint" form such that the optimal $\beta$ would be the same.

$\endgroup$
2
  • $\begingroup$ Thanks so much for the answer. So it is the absolute value of the weight that is penalized to zero. So the true values of the weights can be negative, right? $\endgroup$ Jul 14, 2018 at 10:58
  • $\begingroup$ @Haobing Chu Yes. $\endgroup$ Jul 14, 2018 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.