41
$\begingroup$

I understand what role lambda plays in an elastic-net regression. And I can understand why one would select lambda.min, the value of lambda that minimizes cross validated error.

My question is Where in the statistics literature is it recommended to use lambda.1se, that is the value of lambda that minimizes CV error plus one standard error? I can't seem to find a formal citation, or even a reason for why this is often a good value. I understand that it's a more restrictive regularization, and will shrink the parameters more towards zero, but I'm not always certain of the conditions under which lambda.1se is a better choice over lambda.min. Can someone help explain?

Improve this question
$\endgroup$
2

2 Answers 2

Reset to default
42
$\begingroup$

Friedman, Hastie, and Tibshirani (2010), citing The Elements of Statistical Learning, write,

We often use the “one-standard-error” rule when selecting the best model; this acknowledges the fact that the risk curves are estimated with error, so errs on the side of parsimony.

The reason for using one standard error, as opposed to any other amount, seems to be because it's, well... standard. Krstajic, et al (2014) write (bold emphasis mine):

Breiman et al. [25] have found in the case of selecting optimal tree size for classification tree models that the tree size with minimal cross-validation error generates a model which generally overfits. Therefore, in Section 3.4.3 of their book Breiman et al. [25] define the one standard error rule (1 SE rule) for choosing an optimal tree size, and they implement it throughout the book. In order to calculate the standard error for single V-fold cross- validation, accuracy needs to be calculated for each fold, and the standard error is calculated from V accuracies from each fold. Hastie et al. [4] define the 1 SE rule as selecting the most parsimonious model whose error is no more than one standard error above the error of the best model, and they suggest in several places using the 1 SE rule for general cross-validation use. The main point of the 1 SE rule, with which we agree, is to choose the simplest model whose accuracy is comparable with the best model.

The suggestion is that the choice of one standard error is entirely heuristic, based on the sense that one standard error typically is not large relative to the range of $\lambda$ values.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thank you! Now I can finally cite something appropriate when the question comes up for those unfamiliar with the "standard" choice of lambda. The link to Krstajic et al looks great, too. $\endgroup$
    – jhersh
    Commented Feb 20, 2015 at 22:23
  • $\begingroup$ That citation only says "1se was found to be optimal for classification". But the question asked about regression! There are alternatives. If we try e.g. stepping back to 2se, we get the problem that lambda is too large and shrinks the coeffts too much. But we could e.g. rebuild the model which excludes all variables not selected at lambda.1se in the original model. $\endgroup$
    – smci
    Commented Feb 24, 2017 at 0:42
  • $\begingroup$ @smci which citation? It's not in either of the quotes I extracted, which both suggest that the 1-SE rule is applicable in general, not just in classification. $\endgroup$
    – shadowtalker
    Commented Feb 24, 2017 at 0:56
11
$\begingroup$

Breiman et al.'s book (cited in the other answer's quote from Krstajic) is the oldest reference I've found for the 1SE rule.

This is Breiman, Friedman, Stone, and Olshen's Classification and Regression Trees (1984). They "derive" this rule in section 3.4.3.

So if you need a formal citation, that seems to be the original source.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.