7
$\begingroup$

In the documentation to R's glmnet package it states that when fitting an elastic net, the glmnet function will use a series of $\lambda$ values starting at the smallest $\lambda$ for which all coefficients are zero. How can I find such a value of $\lambda$?

$\endgroup$
3
  • $\begingroup$ are you asking how to find $\lambda$ mathematically or how to find it in glmnet model? $\endgroup$
    – Haitao Du
    Commented Jul 18, 2017 at 16:13
  • $\begingroup$ Mathematically. $\endgroup$
    – badmax
    Commented Jul 19, 2017 at 15:19
  • $\begingroup$ Related: stats.stackexchange.com/questions/166630/… $\endgroup$
    – Henry
    Commented May 28, 2019 at 20:40

2 Answers 2

5
$\begingroup$

A lasso solution $\widehat{\beta}(\lambda)$ solves $$\min_\beta \frac{1}{2}||y-X\beta||_2^2 +\lambda||\beta||_1.$$ and it is well known that we have $\widehat{\beta}(\lambda)=0$ for all $\lambda \geq \lambda_1$ where $\lambda_1 = \max_j |X_j^Ty|$, which should give you the desired value.

Note that $\lambda_1$ may need a different scaling if the objective function is scaled differently.


Using the cars example with GLMNET:

fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1], intercept=FALSE, standardize=FALSE) 1/32*max(abs(t(as.matrix(mtcars[,-1]))%*%mtcars[,1]))/(head(fit$lambda))[1]

This gives the value 1, as expected.

Note that standardize as well as intercept is set to FALSE. If standardize and intercept is set to TRUE, then the value of $\lambda$ is calculated on the scaled regressors. (In this regards, take a look at https://think-lab.github.io/d/205/#5 for how to perform a proper scaling to get the results you want.):

xy<-scale(mtcars) fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1]) (1/32*max(abs(t(xy[,-1])%*%mtcars[,1]*sqrt(32/31))))/(head(fit$lambda))[1]

This once again gives the value 1...

However I am not sure what glmnet is calculating if intercept = TRUE but standardize = FALSE.


We saw that glmnet with its standard options calculates $\lambda_{1}$ as $$\lambda_{1} = \max_j| \frac{1}{n} \sum_{i=1}^n x_j^*y|$$, where $x_j^* = \frac{x_j-\overline{x_j}}{\sqrt{\frac{1}{n}\sum_{i=1}^n (x_j-\overline{x_j})^2}}.$

It turns out that for an elastic net problem (corresponding to $\alpha \in (0,1]$ in glmnet) its maximum value $\lambda_{1,\alpha}$ is calculated as

$$\lambda_{1,\alpha}= \lambda_{1}/\alpha$$.

Indeed, setting for example $\alpha=0.3$ we have:

aa<-0.3 xy<-scale(mtcars) fit<-glmnet(as.matrix(mtcars[,-1]),mtcars[,1],a=aa) 1/aa*(1/32*max(abs(t(xy[,-1])%*%mtcars[,1]*sqrt(32/31))))/(head(fit$lambda))[1]

which results once again in an output value of $1$.

That's for the calculations. Note however that the elastic net criterion can be rewritten as a standard lasso problem.

$\endgroup$
6
  • $\begingroup$ we have: (1/32*max(abs(t(xy[,-1])%*%xy[,1]*(sd(mtcars[,1])*sqrt(32/31))))) == (1/32*max(abs(t(xy[,-1]*sqrt(32/31))%*%mtcars[,1])))) and can conclude that, standardize does only standardize the independent variables s.t. $\sum_{i=1}^{n}x_{ij}^2=n$ but the option doesn't touch the dependent variable. $\endgroup$
    – BloXX
    Commented Jul 18, 2017 at 17:26
  • 2
    $\begingroup$ The question seems to be asking about the case of elastic net. Perhaps it would be worthwhile to include the additional step of the derivation for finding this smallest $\lambda$ in the elastic net case, since this question would be a duplicate otherwise. Also, due to the scaling you use in the lasso definition, it seems like the result should actually be that $\lambda_1=2 \max_j |x_j^T y|$ for the lasso. $\endgroup$
    – user795305
    Commented Jul 19, 2017 at 6:28
  • $\begingroup$ Thank you for pointing out my mistake - I adjusted the objective function by the term $\frac{1}{2}$ instead adjusting the $\lambda$. Besides I added the calculations for the elastic net. Note, that $\alpha \to 0$ or $\alpha= 0$ is a special case (which would result in $\lambda = \infty$). Instead glmnet does its calculations for $\alpha = 0.001$. see its help-file. $\endgroup$
    – chRrr
    Commented Jul 19, 2017 at 13:46
  • $\begingroup$ "elastic net criterion can be rewritten as a standard lasso problem." How? $\endgroup$ Commented Jul 19, 2017 at 14:16
  • 1
    $\begingroup$ You just have to augment your data - take a look at Lemma 1 of "Regularization and variable selection via the elastic net" by Zou and Hastie for more details. $\endgroup$
    – chRrr
    Commented Jul 19, 2017 at 15:30
1
$\begingroup$

First, I think glmnet will start with a large $\lambda$ instead of a small $\lambda$. Here is the documentation: note, if we want to specify $\lambda$, it is better in decreasing order.

Typical usage is to have the program compute its own lambda sequence based on nlambda and lambda.min.ratio. Supplying a value of lambda overrides this. WARNING: use with care. Do not supply a single value for lambda (for predictions after CV use predict() instead). Supply instead a decreasing sequence of lambda values. glmnet relies on its warms starts for speed, and its often faster to fit a whole path than compute a single fit.

Also, See my question here: Why does `R` `glmnet` need to run with $\lambda$ in decreasing order?


The fitting results contains the lambda value used. Here is an example.

library(glmnet)
fit=glmnet(as.matrix(mtcars[,-1]),mtcars[,1])
head(fit$lambda)
[1] 5.146981 4.689737 4.273114 3.893502 3.547614 3.232454
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.