Why does Group Lasso use L2 norm for individual group penalties?

Question

In group lasso $$\min_{\beta}\left\{\frac{1}{2} \left\lVert{y}-\sum_{l=1}^mX^{(l)}{\beta^{(l)}}\right\rVert_2^2 +\lambda\sum_{l=1}^m\sqrt{p_l}\left\lVert{\beta^{(l)}}\right\rVert_q\right\}$$ the individual group penalties use L2 norm, that is $q = 2$. What's the intuition for this choice? Would the main property of group lasso (that entire groups may be eliminated) still hold if $q$ was set to 1 or some other value?

(I assume the presence of group size weighting $\sqrt{p_l}$ makes no difference for the intuition, and for group elimination property; I included it only because it seems to be the standard formulation.)

Answer 1

Any $q > 1$ would define an estimator which performs group-wise selection and is the minimizer of a convex function. When $q=1$, the estimator reduces to a (weighted) lasso which does not perform group-wise selection. When $q < 1$, the objective function is non-convex.

In the original paper,

Yuan, M., & Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1), 49-67.

the motivation is given (in figure 1) that the penalty looks like the lasso's penalty for coefficients in different groups while looking like the ridge regression's penalty for coefficients within the same group. This suggests that an explanation for why the group lasso uses $q=2$ reduces to an explanation for the utility of ridge regression over other $\ell_q$ penalized regression estimators. The standard intuition for this is that in ridge regression the coefficients are treated as being neutrally, in that they are neither being encouraged to be nearly sparse (as when $q < 2$) or to be nearly equal to each other (as when $q > 2$).

A more practical explanation for why $q=2$ could just be that the statistical community is comfortable with $\ell_2$ penalization and that the choice $q=2$ made possible deriving a LARS-type algorithm for fast computation. In the time when this paper was published, it was standard for new convex penalized regression estimators to be accompanied with a LARS-type algorithm.

---

Fig. 1.:

Answer 2

Where did you read that it uses q = 2? Here, q should be the number of variables in the jth group. If each covariate is its own group of size 1, then the group lasso reduces to the lasso. If they are all treated as a single large group, it reduces to ridge regression.

The objective function for the group lasso is more properly written as $\left\| y - \sum _ { g = 1 } ^ {G } X _ { g } \beta _ { g } \right\| _ { 2 } ^ { 2 } \ + \ \lambda \sum _ {g = 1 } ^ { G } \left\| \beta _ { g } \right\| _ { q_ { g } }$.

Note the use of letters here is arbitrary of course. I like to use G/g for designating the group terms.

Basically the penalty term reduces to the L2 norm on the q coefficients within the gth group, so it doesn't perform variable selection within the groups. However, the penalty term ends up being the sum of the norms for each group, so this is just the same as the lasso penalty having p predictors in its penalty term, $\sum_{p = 1}^P \left\| \beta_p \right\|^{1/P}$. Hence you can see why the group lasso reduces to the lasso if the number of groups G = p.

The sparse group lasso on the other hand can perform variable selection within groups.

$\left\| y - \sum _ { g = 1 } ^ {G } X _ { g } \beta _ { g } \right\| _ { 2 } ^ { 2 } \ \ + \ \ (1 - \alpha) \lambda \sum _ {g = 1 } ^ { G } \left\| \beta _ { g } \right\| _ { q_ { g }} \ \ + \ \ \alpha \lambda\sum_{p = 1}^P \left\| \beta_p \right\|^{1/P}$

This requires the determination of an additional parameter $\alpha$ which controls the balance of individual-parameter L1 penalties and the group penalty. This makes it extremely similar to the elastic net. If you let G = 1 such that all parameters are part of the same group, this reduces to the elastic net just as the group lasso reduces to the ridge when G = 1. And if each coefficient is its own group such that G = p I suppose you'd have a double lasso!