3
$\begingroup$

I assumed that cv.glmnet works as follows:

  1. Generate multiple glmnet fits for the entire data, presumably for automated lambda sequence using coordinate descent
  2. Use the lambdas gotten in step 1, and find rmse for different cross validation folds, again using glmnet
  3. The lambda with the best RMSE across folds is chosen as lambda_min
  4. Return the overall model (in step1) corresponding to lambda_min

But the github code (https://github.com/cran/glmnet/blob/master/R/cv.glmnet.R) suggests that the different cv folds use their own lambda sequence. Why was this change done in cv.glmnet code ? How does it add up ?

I looked at the web.stanford.edu/~hastie/glmnet/glmnet_alpha.html and it seems that we can compute a elastic model for any specified lambda within the range of lambda sequence used, once the model has been computed. So, while we use appropriate warm starts for each cross validation fold, but the original lambda sequence only acts as the lambda grid from which we choose the perfect lambda. Is my interpretation correct ?

$\endgroup$
  • $\begingroup$ The heuristics used to construct the lambda sequence are based on the training data. In CV, you're explicitly using different training data for each fold, so the heuristics will produce different results. $\endgroup$ – Sycorax Feb 12 at 19:00
  • $\begingroup$ But do we not need to find the cross validation error for the lambda sequence found in step 1 ? And cv error is nothing but a lasso model with that lambda ? Let say y1, y2, y3 be the lambdas obtained in step1. Now we to pick one of them by cross validation. So for each lambda in (y1, y2, y3), we build glmnet model for each cross validation fold, and find average rmse. The lambda that gives the minimum rmse should be chosen. say y2. The model corresponding to y1 (in step 1) should be returned $\endgroup$ – cryptickey Feb 12 at 19:11
  • $\begingroup$ I don't understand your comment. Why would y1 be the correct answer if y2 has the lowest error? $\endgroup$ – Sycorax Feb 12 at 19:49
  • $\begingroup$ Sorry for the typo.. i meant y2 only..But then why is a new lambda sequence being used in every fold, as we need to compute cv errors for a set of lambda only $\endgroup$ – cryptickey Feb 13 at 5:59
  • $\begingroup$ I looked at the web.stanford.edu/~hastie/glmnet/glmnet_alpha.html and it seems that we can compute a elastic model for any specified lambda within the range of lambda sequence used, once the model has been computed. So, while we use appropriate warm starts for each cross validation fold, but the original lambda sequence only acts as the lambda grid from which we choose the perfect lambda. Is it correct ? $\endgroup$ – cryptickey Feb 18 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.