1
$\begingroup$

I have a question about the elastic net penalty as implemented in glmnet in R compared to the original paper by Zou and Hastie (2005). In glmnet the penalty is listed as

$$ (1-\alpha)/2||\beta||_2^2+α||\beta||_1. $$

but in the paper it is

$$(1-\alpha)||\beta||_1 + \alpha||\beta||_2^2. $$

Does anyone know where the factor $\frac12$ some from? (Never mind the fact that the $\alpha$'s were swapped between the two parameterisations.) In both cases the penalties are multiplied by $\lambda$, but what are the mathematical/technical arguments for not using a simple convex combination of the lasso and ridge penalties?

$\endgroup$
  • $\begingroup$ There's definitely no mathematical argument, because the first formulation can be converted to the second simply by a change of units of measure for the betas (which amounts to a change in how the columns of the design matrix are standardized). $\endgroup$ – whuber Aug 4 '16 at 22:31
4
$\begingroup$

Both of these are simple convex combinations of lasso and ridge penalities, only the meaning of ridge penalty is slightly different in each.

In the first, the ridge penalty term is taken to be

$$ \frac{1}{2} \left| \beta \right|_2^2 $$

and in the second the ridge penalty is taken to be

$$ \left| \beta \right|_2^2 $$

The way its written as $ \frac{\alpha - 1}{2} \left| \beta \right|_2^2 $ is, unfortunately, a little confusing. You should think of the $\frac{1}{2}$ as being part of the penalty and not part of the convex combination.

As for why the $\frac{1}{2}$ at all? In certain calculations (for example, when deriving the update step in glmnet) you need to take a gradient with respect to $\beta$. The $\frac{1}{2}$ is mathematically convenient to have, as it cancels with the exponent of $2$ after differentiation. Since it doesn't hurt anything conceptually to include the $\frac{1}{2}$, many people do.

$\endgroup$
  • $\begingroup$ Ah that makes sense - I just couldn't figure out why it was a good idea to specify it explicitly. Thanks to both you and @whuber $\endgroup$ – MsBaffled Aug 4 '16 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.