0
$\begingroup$

In group lasso we try to minimize the following $$\frac{1}{2}\|Y - \sum_{i=1}^JX_j\beta_j\|^2 + \lambda\sum_{i=1}^J\|\beta_j\| $$ Often they change this to $$\frac{1}{2}\|Y - \sum_{i=1}^JX_j\beta_j\|^2 + \lambda\sum_{i=1}^J\sqrt{p_j}\|\beta_j\|, $$ with $p_j$ the number of variables in group $j$. The reason they give is that 'since all groups are equally penalized in the firsts formula, larger groups would be more likely to be selected'. Can someone explain this reasoning?

$\endgroup$
1
$\begingroup$

The logic is that you could have 2 categorical variables, the first has 101 levels and the second only has 2. When you represent these in the linear model you use a series of indicator variables for each of the levels except a baseline level. This means the first variable has 100 binary variables and the second has only 1. In this extreme example you would be penalizing the two features in a highly unequal fashion-you want to have the two features weighted equally, multiplying by $p_i$ in this case (100,1) puts both features on an equal footing when they are penalized by $\lambda$. Otherwise $\lambda$ knows nothing about the 101 binary variables in the model and penalizes them all the same.

$\endgroup$
  • $\begingroup$ And why thus this imply that if we don't do this, the large group is more likely to be selected? I would just think that since they are with so much, they have a large weight in the function that you want to minimize, so they are removed the first... $\endgroup$ – Chris Nov 12 '17 at 20:39
  • $\begingroup$ The point is if any of the 100 indicators in the first feature are chosen then the first feature is chosen, whereas with the second feature you need to choose this single binary indicator variable. After weighting for group size, the binary indicators are removed according to the same criteria as under the regular lasso model. $\endgroup$ – Lucas Roberts Nov 12 '17 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.