Which lambda is cv.glmnet solving for?

This is my understanding of glmnet:

if OLS is minimizing RSS, where

$RSS = \sum(y-\beta x)^2$

I believe glmnet is minimizing:

$RSS - \sum(\alpha |\beta_j| + (1-\alpha) \beta_j^2)$ where $\alpha=\lambda_1/(\lambda_1+\lambda_2)$

$\lambda_1$ and $\lambda_2$ come from lasso and ridge regression, but I'm confused if $\lambda_1 = \lambda_2$ such that cv.glmnet in glmnet package of R is solving for a single variable (along the whole path) $\lambda$? But then $\alpha = 0.5$ always.

If $\lambda_1 = \lambda_2$, is the glmnet penalty equivalent to $RSS - \lambda |\beta| - \lambda \beta^2$

I've read through Hastie et al. 2009 Elements of Statistical Learning and Zou and Hastie 2005 so now I'm trying to get some clarification on the lambdas and alpha. Thanks

EDIT:

I found this to be a useful formulation in Friedman et al (2010) Regularization Paths for Generalized Linear Models via Coordinate Descent.

$$1/2N * \sum (y_i - \beta_0 - X\beta)^2 + \lambda P_\alpha (\beta)$$ where $$P_\alpha (\beta) = \sum (1/2 (1-\alpha) \beta_j^2 + \alpha |\beta_j|)$$ I thought it provided some intuition how lambda and alpha exist together.

• – Sycorax Oct 5 '16 at 7:04
• relevant and helpful, but I still don't understand how a single $\lambda$ comes into the picture when the penalty is formulated with $\alpha$ – Dominik Oct 5 '16 at 16:15
• You haven't correctly given the penalty: the entire term must be multiplied by a second independent parameter, called "$\lambda$" in the documentation. It functions like the $\lambda$ in your edit (and is directly proportional to it). This lambda has nothing to do with your $\lambda_1$ and $\lambda_2$. cv.glmnet helps you find $\lambda$, but you have to specify $\alpha$. – whuber May 26 '17 at 17:21

$\alpha=\frac{\lambda_1}{\lambda_1+\lambda_2} \text{and } 1-\alpha=\frac{\lambda_2}{\lambda_1+\lambda_2}$. And because $\lambda_i\ge0,$ it should be clear that $\alpha\in[0,1].$ So in glmnet, $\lambda=\lambda_1+\lambda_2$, and each penalty has a coefficient that is either $\alpha(\lambda_1+\lambda_2)$ or $(1-\alpha)(\lambda_1+\lambda_2)$.

But treating $\alpha$ independently of $\lambda_1, \lambda_2$ is convenient as a conceptual model because it controls how much of a ridge and lasso penalty is applied, with either extreme arising as a special case. And you can make a model "more lasso" or "more ridge" by adjusting $\alpha$ without having to worry about how to adjust $\lambda_i$ relative to the size of $\lambda_j, j\neq i$. That is, treated separately, $\alpha$ controls the range of elastic net compositions on a continuum of ridge to lasso, while $\lambda$ controls the overall magnitude of the penalty. The two can be thought of as independent model hyper-parameters. The method with two lambdas links the two penalties.

And if both $\lambda_1$ and $\lambda_2$ are 0, that should correspond to no penalty, but the fraction $\frac{\lambda_1}{\lambda_1+\lambda_2}=\frac{0}{0}$ is unsightly and indeterminate.

• Thanks. Could you elaborate your 3rd sentence? how did you get to $\lambda = \lambda_1 + \lambda_2$ ? – Dominik Oct 5 '16 at 18:01
• That's just the definition/convention that glmnet uses. – Sycorax Oct 5 '16 at 18:52
• ok. and when you said "α controls the range of elastic net compositions on a continuum of ridge to lasso, while λ controls the overall magnitude of the penalty.", doesn't $\alpha$ scale $\lambda$ therefore also controlling the magnitude of the penalty? – Dominik Oct 5 '16 at 19:17
• The total magnitude $\lambda$ isn't changed by $\alpha$. $\alpha$ just changes how much penalty is applied to lasso and how much penalty is applied to ridge. Because it's a convex combination, total penalty remains constant even as you change $\alpha$. If $\alpha$ is 0 or 1, you're doing either lasso or ridge regression because the penalty to one is 0. – Sycorax Oct 5 '16 at 19:25
• That's addressed in this thread: stats.stackexchange.com/questions/74542/… – Sycorax Oct 6 '16 at 13:58

This shows that glmnet doesn't cross validate over $\alpha$, so that the cross validation is just one-dimensional, as I think you suspect.