9
$\begingroup$

I'm wondering how to approach properly training and testing a LASSO model using glmnet in R?

  • Specifically, I'm wondering how to do so if a lack of an external test data set necessitates I use cross-validation (or other similar approach) to test my LASSO model.

Let me break down my scenario:

I only have one data-set to inform and train my glmnet model. As a result, I'll have to use cross-validation to split up my data to also generate a way to test my model.

I'm already using cv.glmnet, which according to the package details:

Does k-fold cross-validation for glmnet, produces a plot, and returns a value for lambda.

  • Is the cross-validation performed in cv.glmnet simply to pick the best lambda, or is it also serving as a more general cross-validation procedure?

    • In other words, do I still need to do another cross-validation step to "test" my model?

I'm working with the assumption that, "yes I do."

That being the case, how do I approach cross validating my cv.glmnet model?

  • Do I have to do so manually, or is perhaps the caret function useful for glmnet models?

  • Do I use two concentric "loops" of cross validation?... Do I use an "inner loop" of CV via cv.glmnet to determine the best lambda value within each of k folds of an "external loop" of k-fold cross validation processing?

    • If I do cross-validation of my already cross-validating cv.glmnet model, how do I isolate the "best" model (from the "best" lambda value) from each cv.glmnet model within each fold of my otherwise "external loop" of cross validation?

      • Note: I'm defining "best" model as the model associated with a lambda that produces an MSE within 1 SE of the minimum ... this is the $lambda.1se in the cv.glmnet model.

Context:

I'm trying to predict tree age ("age") based on tree diameter ("D"), D^2, and species ("factor(SPEC)"). [resulting equation: Age ~ D + factor(SPEC) + D^2]. I have ~50K rows of data, but the data is longitudinal (tracks individuals through time) and consists of ~65 species.

$\endgroup$

migrated from stackoverflow.com Oct 8 '16 at 1:22

This question came from our site for professional and enthusiast programmers.

  • 1
    $\begingroup$ For those who voted to close without commenting: that's not helpful...let me know what your issue is in the comments and I'll try to fix it. $\endgroup$ – theforestecologist Oct 7 '16 at 23:52
  • 1
    $\begingroup$ They aren't voting to close they are voting to migrate it to CrossValidated. I just added my vote to that. $\endgroup$ – Hack-R Oct 8 '16 at 0:00
  • $\begingroup$ @theforestecologist: You should be able to see the reasons being cited for closure/migration by clicking on the 'close'-button. $\endgroup$ – DWin Oct 8 '16 at 0:38
  • $\begingroup$ Thanks for asking, I had exactly this question. And I can't use caret because i have multivariate Y. But have you inspected the source code and confirmed that no additional CV is needed? The source code can be quite difficult to follow. $\endgroup$ – qoheleth Mar 29 '17 at 1:27
3
$\begingroup$

Is the cross-validation performed in cv.glmnet simply to pick the best lambda, or is it also serving as a more general cross-validation procedure?

It does almost everything needed in a cross-validation. For example, it fits possible lambda values on the data, chooses the best model and finally trains the model with the appropriate parameters.

For example, in the returned object::

cvm is the mean cross-validated error. cvsd is the estimated standard deviation.

Like other returned values, these are calculated on the test set. Finally, the

glmnet.fit gives the model trained on all the data (training + test) with the best parameters.

Do I have to do so manually, or is perhaps the caret function useful for glmnet models?

You need not do this manually. 'Caret' would be very useful, and is one of my favourite package because it works for all the other models with same syntax. I myself often use caret rather than cv.glmnet. However, in your scenario it is essentially the same.

Do I use two concentric "loops" of cross validation?... Do I use an "inner loop" of CV via cv.glmnet to determine the best lambda value within each of k folds of an "external loop" of k-fold cross validation processing?

You could do this and this concept is very similar to the idea of Nested Cross-Validation Nested cross validation for model selection.

If I do cross-validation of my already cross-validating cv.glmnet model, how do I isolate the "best" model (from the "best" lambda value) from each cv.glmnet model within each fold of my otherwise "external loop" of cross validation?

Just run a loop where you generate a training data and test data run cv.glmnet on training data and use the model glmnet.fit to predict on the test data.

$\endgroup$
  • $\begingroup$ @discupulus: Thanks. Could you provide some sort of evidence for that? (i.e., walk me through it please). Also, given your answer, does that mean that no further cross validation processes are necessary to report a performance metric for the data? (I could simply report the MSE of the lambda.1se case as my final model performance?) $\endgroup$ – theforestecologist Oct 8 '16 at 4:06
  • $\begingroup$ Yes, no further cross-validation is necessary. For the evidence, you can look at the source code of cv.glmnet function as R is opensource. Just type cv.glmnet in console. $\endgroup$ – discipulus Oct 8 '16 at 4:11
  • 3
    $\begingroup$ @discipulus. I emailed Trevor Hastie asking "does cv.glmnet (R implementation) only do CV to choose lambda? or does it also do an outer CV to validate the chosen lambda? In other words, do we have to code our own outer CV if we want to validate the chosen lambda?" and he replied (rather quickly) "Yes, just to pick lambda", which I interpret to mean it only does the inner CV, and we will have to code our own outer CV. $\endgroup$ – qoheleth Apr 11 '17 at 6:26
  • $\begingroup$ @theforestecologist I am trying to learn more about cross-validation and find your post educational. I don't understand what you mean by cv.glmnet also serving as a more general cross-validation procedure. I thought the only parameter available to choose is lambda - what outer layer of cross-validation exists? Would be grateful if you could reply. Thanks! $\endgroup$ – user2450223 Apr 24 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.