I have a question concerning the one-standard-error rule when doing leave-one-out cross-validation.

firstly, my summary of the one-standard-error rule:

When using cross-validation or validation the model with the lowest test error will most likely change for every time the model is fitted with another random assignment of observations to the folds. Therefore, the one-standard-error rule can be used to select the simplest model that is within one standard error of the model with the lowest test error. The rational behind the rule is that, when there is, from a statistical point of view, not much difference between the models, why not choose the simplest model

Secondly, my question:

If the above "definition" holds, then there should be no standard error when applying leave-one-out cross-validation. The reason being that, the observations assigned to the folds are always the same, so there is no variance in the cross-validation error. Therefore, I wonder why RStudio and Matlab still give standard errors when LOOCV is applied?

  • $\begingroup$ What is your basis for believing "the observations assigned to the folds are always the same"? $\endgroup$ – whuber Oct 29 '19 at 15:24
  • 1
    $\begingroup$ @whuber: if LOOCV is applied there is only one possible division, i.e., one observation per fold $\endgroup$ – Arjan Dexters Oct 29 '19 at 16:18
  • $\begingroup$ Is the software comprehensively leaving each observation out or randomly selecting them? (I imagine the best way to find out would be to consult the documentation for whatever functions you are using.) $\endgroup$ – whuber Oct 29 '19 at 16:19

Leave-one-out does have a standard deviation and also a standard error: each of the $n$ folds returns an error estimate, so you can calculate mean as well as standard deviation and standard error over the $n$ fold estimates.

You are right though, that LOO is exhaustive: it is not possible to obtain other than those $n$ surrogate models from an LOO scheme.

What you cannot do in LOO (as opposed to resampling schemes that leave out more than 1 case for each surrogate model, whether they are run exhaustively or not) is separating variance stemming from model instability from variance stemming from the tested data, as LOO confounds those influencing factors. Which might be important information e.g. for model auto-tuning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.