8
$\begingroup$

The lasso and the elastic net are not able to handle variables with more than two categories and therefore a split of categorical variables into dummies is necessary for the application of these methods. This can result in several problems and therefore there exist extensions for the lasso to the group lasso or the sparse group lasso.

However, I am wondering if such extensions also exist for elastic net. Unfortunately, I wasn't able to find any statistical literature about the topic.

Question: Does a group elastic net exist?

$\endgroup$
7
  • 1
    $\begingroup$ Look at R's glmnet package ... $\endgroup$ Aug 7 '17 at 13:22
  • 1
    $\begingroup$ Yes, I think that is correct. $\endgroup$ Aug 7 '17 at 13:49
  • 1
    $\begingroup$ In a very real sense, this "group elastic net" is just a version of "group lasso" where the groups are allowed to overlap. For instance, if $\mathcal{G}$ is your set of groups, then run group lasso on $\mathcal{G} \cup \{ \{1, \dots, p\} \}$, where we consider there to be $p$ features. This will be equivalent to the group elastic net up to a reparameterization of the tuning parameter controlling $\{1, \dots, p\}$. $\endgroup$
    – user795305
    Aug 7 '17 at 15:15
  • 2
    $\begingroup$ The set $\mathcal{G} \cup \{\{1, \dots, p\}\}$ is no longer a partition, unlike $\mathcal{G}$ itself. (This is the overlapping comment.) The part about the different parameterization just relates to the objective function I discuss being a reparameterization of the one that you likely are discussing. This comment can largely be ignored imo. Also, the procedure @kjetilbhalvorsen recommends does not appear to be correct. The grouping discussed there is for when there's a multivariate response. That's different. However, you can, for instance, use the gglasso package to do this. $\endgroup$
    – user795305
    Aug 7 '17 at 18:48
  • 2
    $\begingroup$ (Note: Don't put a space after the "@", otherwise the user does not get notified.) $\endgroup$
    – user795305
    Aug 7 '17 at 18:48
7
$\begingroup$

Let $\mathcal{G}$ be the grouping that you're interested in; that is, let $\mathcal{G}$ be a partition of $\{1, \dots, p\}$, where we consider there to be $p$ features. With response $y \in \mathbb{R}^n$ and design matrix $X \in \mathbb{R}^{n \times p}$, the group lasso estimator is $$\arg\min_{\beta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2.$$ Applying another squared $\ell_2$ penalty to induce overall shrinkage, we'd get the estimator $$\arg\min_{\beta \in \mathbb{R}^p} \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 + \mu \|\beta\|_2^2.$$ We might call this the "group elastic net". By Lagrangian duality, we can write \begin{align*} \arg\min_{\beta \in \mathbb{R}^p} & \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 + \mu \|\beta\|_2^2 \\ = \, \arg\min_{\beta \in \mathbb{R}^p \, : \, \|\beta\|_2^2 \leq C} & \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 \\ = \, \arg\min_{\beta \in \mathbb{R}^p \, : \, \|\beta\|_2 \leq \sqrt{C}} & \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 \\ = \, \arg\min_{\beta \in \mathbb{R}^p} & \frac{1}{2n} \|y - X \beta \|_2^2 + \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 + \tilde\mu \|\beta\|_2 \\ = \, \arg\min_{\beta \in \mathbb{R}^p} & \frac{1}{2n} \|y - X \beta \|_2^2 + \left( \lambda \sum_{g \in \mathcal{G}} |\mathcal{G}|^{1/2} \|\beta_g\|_2 + \tilde\mu' p^{1/2} \|\beta\|_2 \right), \end{align*} where $\tilde\mu$ is the corresponding dual variable and $\tilde\mu' = p^{-1/2} \tilde\mu$. As we can see, this last expression is a group lasso with "overlapping" groups, since $\mathcal{G} \cup \{1, \dots, p\}$ is no longer a partition. Further, the group $\{1, \dots, p\}$ has a dual variable (or tuning variable) $\tilde\mu$ which is distinct from the dual variable $\lambda$ for the other groups.

This can be optimization problem can be solved using the package gglasso. Reading the section on page 9 of the documentation here will tell you about the gglasso function, which should be used. Note that the argument pmax will have to manually supplied with a last component which will serve as a tuning parameter.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.