3
$\begingroup$

On Wikipedia, it says that: "while if there is only a single group, it reduces to ridge regression" (https://en.wikipedia.org/wiki/Lasso_(statistics)#Group_lasso).

However in group lasso we have norm 2, but in ridge regression we have norm 2 squared. Thanks to norm 2 squared we can obtain closed-form solution for ridge regression. I do not think we can do the same for group lasso with just 1 group.

Am I missing something or the 2 things are actually not equivalent? I am interested in solving group lasso with 1 group efficiently (as part of a bigger problem).

$\endgroup$
  • $\begingroup$ I recommend providing a link to the document you reference. $\endgroup$ – HStamper Mar 8 '19 at 1:33
  • $\begingroup$ I think the Wikipedia argument works. Consider the fact that the parameters within a group are combined to get the group penalties using a quadratic form $z^t K_j z$, whereas the group penalties are combined additively. $\endgroup$ – Ben Bolker Mar 8 '19 at 2:12
  • $\begingroup$ It may make it clearer that the two are equivalent if you phrase things in terms of the constrained optimization rather than the Lagrangian. Optimizing subject to the squared norm being less than eta squared is the same as optimizing subject to the norm being less than eta. $\endgroup$ – guy Dec 1 '19 at 15:27
2
$\begingroup$

To see that group LASSO gives a similar solution to ridge regression upto a square in the penalty function, you need to look at subgradient conditions that characterize the solution of the group LASSO estimator. The best reference for this purpose I think is http://statweb.stanford.edu/~tibs/ftp/sparse-grlasso.pdf

These conditions are equation (4) and (5) in the paper, which are

$$ ||X^{\top}_l (y - \sum_{k\neq l} X_k \hat\beta_k)|| < \lambda \qquad \text{ (1)} \qquad \text{gives you condition when } \hat\beta_l=0 $$

and

$$ \hat\beta_l = \left(X^{\top}_l X_l - \frac{\lambda}{||\hat\beta_l ||} \right) X^{\top}_l r \qquad (2) \qquad \text{gives solution if } \hat\beta_l\neq0$$

where

$$ r = y - \sum_{k\neq l} X_k \hat\beta_k$$

Here l stands for a group index and X can be partitioned to non-overlapping groups. If you have a single group, these conditions boil down to

$$ ||X^{\top} (y - X \hat\beta)|| < \lambda \qquad \text{when } \hat\beta=0$$

and

$$ \hat\beta = \left(X^{\top} X - \frac{\lambda}{||\hat\beta ||} \right) X^{\top} y \qquad \text{otherwise }$$

We can now clearly see why group LASSO with a single group is, in fact, ridge regression with the weighted penalty term. The easiest way to solve group LASSO with a single group would be to use efficient implementations of group LASSO in whatever software you use setting the group index accordingly. Else, you could iterate the last equation a few times until the coefficient vector does not change too much. Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.