How does glmnet in caret choose the values of lambda and how does it compute coefficients of the model?

Question

I have a question that I've been struggling with. My students are asking me, but I can't figure it out myself.

When I train LASSO regression in R caret, I use the method "glmnet" and a grid with a fixed value of $\alpha=1$ and a range of values for $\lambda$. Here is a minimal example (it seems to be to work the same way in all versions of R that I tried and on different operating systems):

lambda <- 10^seq(-1, 0.6, length = 5)
lambda

lasso <- train(
  mpg ~., data = mtcars, method = "glmnet",
  trControl = trainControl("cv", number = 10),
  tuneGrid = expand.grid(alpha = 1, lambda = lambda),
  preProcess = c("scale")
)

lasso

The output looks as follows:

[1] 0.1000000 0.2511886 0.6309573 1.5848932 3.9810717
glmnet 

32 samples
10 predictors

Pre-processing: scaled (10) 
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 29, 29, 28, 30, 28, 29, ... 
Resampling results across tuning parameters:

  lambda     RMSE      Rsquared   MAE     
  0.1000000  2.623500  0.8248033  2.330232
  0.2511886  2.593066  0.8343992  2.285429
  0.6309573  2.425283  0.8775376  2.126263
  1.5848932  2.637736  0.8920448  2.320468
  3.9810717  4.123196  0.8613370  3.685106

Tuning parameter 'alpha' was held constant at a value of 1
RMSE was used to select the optimal model using the smallest value.
The final values used for the model were alpha = 1 and lambda = 0.6309573.

I understand that in order to extract coefficients of the model for different values of $\lambda$, I need to use the function "coef". I do it like this:

coef(lasso$finalModel, lambda)

and I get the intended output:

11 x 5 sparse Matrix of class "dgCMatrix"
                      1           2           3          4          5
(Intercept) 20.12070307 29.64873768 36.40838250 33.3021609 24.3644781
cyl         -0.39267064 -1.09019265 -1.57746798 -1.4822386 -0.4249298
disp         .           .           .           .          .        
hp          -0.89172557 -0.94001668 -0.91663301 -0.3768918  .        
drat         0.41257152  0.26861963  .           .          .        
wt          -2.58105019 -2.54389779 -2.70993457 -2.2109627 -0.8520593
qsec         0.82333186  0.30725294  .           .          .        
vs           0.05920734  0.02024462  .           .          .        
am           1.05461719  0.70458954  0.05087579  .          .        
gear         0.22456562  .           .           .          .        
carb        -0.75029547 -0.50103229 -0.01280360  .          .        

However, here comes something that puzzles me. Suppose that I do not specify the values of $\lambda$ for which I want the coefficients of my model, i.e., run the following command:

coef(lasso$finalModel)

Then R prints a matrix of dimensions $11\times 79$. Here is what it looks like (just first 6 columns):

11 x 79 sparse Matrix of class "dgCMatrix"
   [[ suppressing 79 column names ‘s0’, ‘s1’, ‘s2’ ... ]]
                                                                            
(Intercept) 20.09062 21.63712187 23.2405070 24.7015023 26.0327070 27.2462419
cyl          .       -0.02129088 -0.2585639 -0.4748148 -0.6718548 -0.8520529
disp         .        .           .          .          .          .        
hp           .        .           .          .          .          .        
drat         .        .           .          .          .          .        
wt           .       -0.44789995 -0.6855245 -0.9019950 -1.0992346 -1.2784335
qsec         .        .           .          .          .          .        
vs           .        .           .          .          .          .        
am           .        .           .          .          .          .        
gear         .        .           .          .          .          .        
carb         .        .           .          .          .          .        

It looks like there exist some $79$ values of $\lambda$ for which the LASSO model has been trained. And it's true! These values of $\lambda$ can be extracted from the model:

lasso$finalModel$lambda

outputs

 [1] 5.146981063 4.689737451 4.273114102 3.893502423 3.547614399 3.232454113
 [7] 2.945291799 2.683640194 2.445232995 2.228005236 2.030075391 1.849729089
[13] 1.685404255 1.535677586 1.399252222 1.274946511 1.161683777 1.058482992
[19] 0.964450281 0.878771176 0.800703567 0.729571269 0.664758168 0.605702885
[25] 0.551893910 0.502865176 0.458192020 0.417487504 0.380399064 0.346605460
[31] 0.315813986 0.287757942 0.262194320 0.238901699 0.217678330 0.198340388
[37] 0.180720374 0.164665674 0.150037230 0.136708336 0.124563544 0.113497662
[43] 0.103414843 0.094227753 0.085856819 0.078229536 0.071279840 0.064947535
[49] 0.059177775 0.053920585 0.049130429 0.044765817 0.040788945 0.037165367
[55] 0.033863699 0.030855341 0.028114238 0.025616646 0.023340934 0.021267390
[61] 0.019378053 0.017656561 0.016088000 0.014658787 0.013356541 0.012169982
[67] 0.011088835 0.010103733 0.009206145 0.008388297 0.007643104 0.006964111
[73] 0.006345439 0.005781728 0.005268095 0.004800092 0.004373665 0.003985121
[79] 0.003631093

Here come my two questions:

1)How does my lasso model choose these values of $\lambda$? For instance, why did it decide to vary $\lambda$ but not $\alpha$? And why does it seem to have ignored the values of $\lambda$ but not $\alpha$ that I specified in the "train" function call?

2)How can it compute the coefficients for values of $\lambda$ that are not in this list, e.g, values of $\lambda$ that I specified in the "train" function call?

Answer 1

I was curious myself, this is what i found out:

caret uses Cross validation to choose the optimal value of $\lambda$. After it has found the optimal value of $\lambda$, it fits a LASSO model on the complete dataset. This final model can be used for prediction, and we want the LASSO regression coefficients for this model at the optimal value of $\lambda$. These are the mysterious $\lambda$ values you found.

LASSO as a method (to be more specific, the glmnet() implementation) works with its own internal grid of lambda values (the "regularization path"). See the glmnet() R documentation for how these values are chosen automatically.

caret stores the optimal $\lambda$ value (as determined by CV) as lambdaOpt in the final fit object generated by glmnet(). You can check this yourself in the source code for the caret glmnet method at https://github.com/topepo/caret/blob/master/models/files/glmnet.R

When we use predict() on the caret object, this lambdaOpt value is used with the glmnet() object to create the predictions at the optimal value of $\lambda$. The internal grid of lambda values is used with linear interpolation between the grid values closest to lambdaOpt to calculate the coefficients and make the predictions (if the exact argument of glmnet() is FALSE, which is the default).

Hope this helps.