1
$\begingroup$

This is a basic question about regularization term but I have searched for a while and cannot find the answer. My question is: does Lasso regularization always make some coefficients zero?

A famous explanation figure is something like this: https://www.linkedin.com/pulse/intuitive-visual-explanation-differences-between-l1-l2-xiaoli-chen (FIGURE 3.11 on the top of the linked page)

I can understand most of the plot. However, I am wondering what if the minimum of the whole objective function (RSS + L1), where RSS stands for residual sum of squares, does not happen at the corner but at the edge? I can't see why this is not possible. If this is possible, then we don't have zero coefficients in this case, correct?

To be more precise, the contour of the L1 regularization term is a diamond shape square other than a spherical shape circles. Usually, posts explain the sparsity of coefficients is achieved by L1 regularization because the minimum happens at the vertex of the contour, while anywhere on the circle of the L2 contour is symmetric and hence the coefficients need not to be zero. However, I don't see why the minimum of the whole objective function must happen at the vertex of the diamond. So, can someone show that the minimum is not possible at the edge?

Thank you guys!

edit: Thanks for the link from anonymous Why does the Lasso provide Variable Selection? I think this post solves some of my solution. Basically the solution of the $\beta := \hat{\beta} = (y^Tx - \lambda)/(x^Tx)$ indeed possibly nonzero. However, the key is that $\lambda$ is a hyper-parameter and tuning $\lambda$ is possibly to drive $\hat{\beta}$ zero. Changing $\lambda$ changes the shape of the whole objective function and it is by changing the shape of the objective function that we can set the coefficients zero.

Improve this question
$\endgroup$
5
  • $\begingroup$ Does this answer your question? Why does the Lasso provide Variable Selection? $\endgroup$
    – Samir Rachid Zaim
    Sep 24, 2020 at 13:31
  • $\begingroup$ This is a great question but it has been asked a few different times on the site. Take a look at some of these answers I've included below. Rather than try to replicate the efforts of others, can you try to better explain what about these explanations you do not understand? Stack Exchange - stats.stackexchange.com/questions/45643/… - stats.stackexchange.com/questions/375374/… and in other sites: - towardsdatascience.com/… $\endgroup$
    – Samir Rachid Zaim
    Sep 24, 2020 at 13:34
  • 1
    $\begingroup$ I don't see these posts proved that L1 or Lasso ALWAYS leads to feature number reduction. For example, in the third link, does eq(1.1) and eq(1.2) show that w approaches zero is always the case? Put it another way, is it possible that the minimum happens at the corner of the |beta| contour (diamond shape) not at the edge? $\endgroup$
    – chichi
    Sep 25, 2020 at 9:05
  • 2
    $\begingroup$ Lasso is more than performing one minimization: it explores how the coefficients change as the penalty is modified. It would incorrect to claim that at least one of the coefficients must vanish regardless of the penalty. Indeed, it's clear from looking at any Lasso trace plot you can find that for sufficiently small penalties, all estimated coefficients are nonzero. The claim is that as the penalty increases, almost surely all coefficients (apart from the intercept, which is excluded) eventually vanish. $\endgroup$
    – whuber ♦
    Sep 25, 2020 at 13:03
  • 1
    $\begingroup$ @whuber Thank you whuber I think your comment is spot on! $\endgroup$
    – chichi
    Sep 26, 2020 at 3:33

0

Reset to default