# How to find the smallest $\lambda$ such that all Lasso / Elastic Net coefficients are zero?

In the documentation to R's glmnet package it states that when fitting an elastic net, the glmnet function will use a series of $\lambda$ values starting at the smallest $\lambda$ for which all coefficients are zero. How can I find such a value of $\lambda$?

• are you asking how to find $\lambda$ mathematically or how to find it in glmnet model? Commented Jul 18, 2017 at 16:13
• Mathematically. Commented Jul 19, 2017 at 15:19
• Commented May 28, 2019 at 20:40

A lasso solution $\widehat{\beta}(\lambda)$ solves $$\min_\beta \frac{1}{2}||y-X\beta||_2^2 +\lambda||\beta||_1.$$ and it is well known that we have $\widehat{\beta}(\lambda)=0$ for all $\lambda \geq \lambda_1$ where $\lambda_1 = \max_j |X_j^Ty|$, which should give you the desired value.

Note that $\lambda_1$ may need a different scaling if the objective function is scaled differently.

Using the cars example with GLMNET:

fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1], intercept=FALSE, standardize=FALSE) 1/32*max(abs(t(as.matrix(mtcars[,-1]))%*%mtcars[,1]))/(head(fit$lambda))[1] This gives the value 1, as expected. Note that standardize as well as intercept is set to FALSE. If standardize and intercept is set to TRUE, then the value of$\lambda$is calculated on the scaled regressors. (In this regards, take a look at https://think-lab.github.io/d/205/#5 for how to perform a proper scaling to get the results you want.): xy<-scale(mtcars) fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1]) (1/32*max(abs(t(xy[,-1])%*%mtcars[,1]*sqrt(32/31))))/(head(fit$lambda))[1]

This once again gives the value 1...

However I am not sure what glmnet is calculating if intercept = TRUE but standardize = FALSE.

We saw that glmnet with its standard options calculates $\lambda_{1}$ as $$\lambda_{1} = \max_j| \frac{1}{n} \sum_{i=1}^n x_j^*y|$$, where $x_j^* = \frac{x_j-\overline{x_j}}{\sqrt{\frac{1}{n}\sum_{i=1}^n (x_j-\overline{x_j})^2}}.$

It turns out that for an elastic net problem (corresponding to $\alpha \in (0,1]$ in glmnet) its maximum value $\lambda_{1,\alpha}$ is calculated as

$$\lambda_{1,\alpha}= \lambda_{1}/\alpha$$.

Indeed, setting for example $\alpha=0.3$ we have:

aa<-0.3 xy<-scale(mtcars) fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1],a=aa) 1/aa*(1/32*max(abs(t(xy[,-1])%*%mtcars[,1]*sqrt(32/31))))/(head(fit$lambda))[1] which results once again in an output value of$1$. That's for the calculations. Note however that the elastic net criterion can be rewritten as a standard lasso problem. • we have: (1/32*max(abs(t(xy[,-1])%*%xy[,1]*(sd(mtcars[,1])*sqrt(32/31))))) == (1/32*max(abs(t(xy[,-1]*sqrt(32/31))%*%mtcars[,1])))) and can conclude that, standardize does only standardize the independent variables s.t.$\sum_{i=1}^{n}x_{ij}^2=n$but the option doesn't touch the dependent variable. Commented Jul 18, 2017 at 17:26 • The question seems to be asking about the case of elastic net. Perhaps it would be worthwhile to include the additional step of the derivation for finding this smallest$\lambda$in the elastic net case, since this question would be a duplicate otherwise. Also, due to the scaling you use in the lasso definition, it seems like the result should actually be that$\lambda_1=2 \max_j |x_j^T y|$for the lasso. Commented Jul 19, 2017 at 6:28 • Thank you for pointing out my mistake - I adjusted the objective function by the term$\frac{1}{2}$instead adjusting the$\lambda$. Besides I added the calculations for the elastic net. Note, that$\alpha \to 0$or$\alpha= 0$is a special case (which would result in$\lambda = \infty$). Instead glmnet does its calculations for$\alpha = 0.001$. see its help-file. Commented Jul 19, 2017 at 13:46 • "elastic net criterion can be rewritten as a standard lasso problem." How? Commented Jul 19, 2017 at 14:16 • You just have to augment your data - take a look at Lemma 1 of "Regularization and variable selection via the elastic net" by Zou and Hastie for more details. Commented Jul 19, 2017 at 15:30 First, I think glmnet will start with a large$\lambda$instead of a small$\lambda$. Here is the documentation: note, if we want to specify$\lambda$, it is better in decreasing order. Typical usage is to have the program compute its own lambda sequence based on nlambda and lambda.min.ratio. Supplying a value of lambda overrides this. WARNING: use with care. Do not supply a single value for lambda (for predictions after CV use predict() instead). Supply instead a decreasing sequence of lambda values. glmnet relies on its warms starts for speed, and its often faster to fit a whole path than compute a single fit. Also, See my question here: Why does R glmnet need to run with$\lambda$in decreasing order? The fitting results contains the lambda value used. Here is an example. library(glmnet) fit=glmnet(as.matrix(mtcars[,-1]),mtcars[,1]) head(fit$lambda)
[1] 5.146981 4.689737 4.273114 3.893502 3.547614 3.232454