Elastic net penalty

Question

I have a question about the elastic net penalty as implemented in glmnet in R compared to the original paper by Zou and Hastie (2005). In glmnet the penalty is listed as

$$ (1-\alpha)/2||\beta||_2^2+α||\beta||_1. $$

but in the paper it is

$$(1-\alpha)||\beta||_1 + \alpha||\beta||_2^2. $$

Does anyone know where the factor $\frac12$ some from? (Never mind the fact that the $\alpha$'s were swapped between the two parameterisations.) In both cases the penalties are multiplied by $\lambda$, but what are the mathematical/technical arguments for not using a simple convex combination of the lasso and ridge penalties?

Answer 1

Both of these are simple convex combinations of lasso and ridge penalities, only the meaning of ridge penalty is slightly different in each.

In the first, the ridge penalty term is taken to be

$$ \frac{1}{2} \left| \beta \right|_2^2 $$

and in the second the ridge penalty is taken to be

$$ \left| \beta \right|_2^2 $$

The way its written as $ \frac{\alpha - 1}{2} \left| \beta \right|_2^2 $ is, unfortunately, a little confusing. You should think of the $\frac{1}{2}$ as being part of the penalty and not part of the convex combination.

As for why the $\frac{1}{2}$ at all? In certain calculations (for example, when deriving the update step in glmnet) you need to take a gradient with respect to $\beta$. The $\frac{1}{2}$ is mathematically convenient to have, as it cancels with the exponent of $2$ after differentiation. Since it doesn't hurt anything conceptually to include the $\frac{1}{2}$, many people do.