"Convergence for 1st lambda value not reached"-error using GLMNET package and specifying lambda parameter

Question

I get a weird problem when I specify lambda in the function glmnet, that does not appear if I let the function go through all the lambdas.

When I use the default lambda sequence, it works great:

glmnet.out = glmnet(y=yy[train_id], x=xx[train_id,], family="binomial", alpha=1)

Warning message:
from glmnet Fortran code (error code -82); Convergence for 82th lambda 
 value not reached after maxit=100000 iterations; solutions for larger 
 lambdas returned 

glmnet.out$lambda
[1] 2.126060e-02 1.937187e-02 1.765092e-02 1.608287e-02 1.465411e-02 
    1.335228e-02 1.216610e-02 1.108530e-02 1.010051e-02 9.203207e-03
[11] 8.385619e-03 7.640664e-03 6.961888e-03 6.343413e-03 5.779882e-03 
     5.266413e-03 4.798560e-03 4.372269e-03 3.983849e-03 3.629934e-03

and more rows, up to the 82th value. The warning is OK, as I don't use lambdas that small. The cv.glmnet gives a lambda around 0.001.

Whereas when I specify a single lambda value :

glmnet.out = glmnet(y=yy[train_id], x=xx[train_id,], family="binomial", 
                    alpha=1, lambda=3.629934e-03)  

then I get the following warning message:

Warning messages:
1: from glmnet Fortran code (error code -1); Convergence for 1th lambda 
   value not reached after maxit=100000 iterations; solutions for larger 
   lambdas returned 
2: In getcoef(fit, nvars, nx, vnames) :an empty model has been returned; 
   probably a convergence issue

Note that the 1st lambda value is infinity, so nothing is returned.

How is this possible? I can get around the problem by not specifying lambda, and then manually pick the estimates for prediction with the cross validation dataset like this:

coefficients = as.matrix(coef(glmnet.out)[,selected_lamdda])
xBeta        = cbind(rep(1,length(cv_id)),xx[cv_id,])%*%as.matrix(coefficients)
prediction   = 1/(1+exp(-xBeta))

but it is quite inefficient, as the glmnet does a lot of unnecessary calculations.

Answer 1

From the glmnet R help page:

lambda: A user supplied ‘lambda’ sequence. Typical usage is to have the program compute its own ‘lambda’ sequence based on ‘nlambda’ and ‘lambda.min.ratio’. Supplying a value of ‘lambda’ overrides this. WARNING: use with care. Do not supply a single value for ‘lambda’ (for predictions after CV use ‘predict()’ instead). Supply instead a decreasing sequence of ‘lambda’ values. ‘glmnet’ relies on its warms starts for speed, and its often faster to fit a whole path than compute a single fit.

Try predicting from cv.glmnet instead of glmnet.

Answer 2

This problem happens because you're not using predict; instead, you're attempting to train a model just using one lambda value. The glmnet documentation (?glmnet) strongly advises against this.

lambda: A user supplied lambda sequence. Typical usage is to have the program compute its own lambda sequence based on nlambda and lambda.min.ratio. Supplying a value of lambda overrides this. WARNING: use with care. Do not supply a single value for lambda (for predictions after CV use predict() instead). Supply instead a decreasing sequence of lambda values. glmnet relies on its warms starts for speed, and its often faster to fit a whole path than compute a single fit.

You don't need to compute a new model to get predictions. Just use the predict function, specifying which lambda value to use for the predictions.