I use logistic regression for a binary prediction problem and use LOOCV to estimate the models' prediction accuracies for different predictor sets. A reviewer wants me to use k-fold CV instead. Does this make sense? I would think LOOCV would dominate k-fold CV, and should be preferred unless it it too computationally demanding. Isn't it better to estimate the sub-models on larger samples?
I would think LOOCV would dominate k-fold CV, and should be preferred unless it it too computationally demanding.
Isn't it better to estimate the sub-models on larger samples?
We expect the sub-models to be more similar to the model (i.e. the model trained on the full data set) in many cases, leading to lower bias, yes.
OTOH, LOO will always (systematically) test with a case that is from a class that is underrepresented compared to the training set of the model. For training algorithms that estimate something along the lines of relative class frequency from the training data, this can lead to a surprisingly large pessimistic bias in the LOO estimate.
With a cross validation scheme that leaves out more cases, the problem is less severe, and can even be deliberately counteracted by stratified k-fold CV. Whether this is a sensible choice or not of course depends on the situation at hand.
In addition, k-fold CV can be repeated (aka iterated) with new random partitions.
Doing this has two advantages: You can average out the effect of model instabiliy (leading to lower variance uncertainty on the estimate), and (IMHO even more importantly) you can separate the effects of having a finite test sample size from the effects of model instability. This is not possible with LOO, since LOO has perfect correlation between left out sample and surrogate model.
So, doing a standard 10-fold CV instead of LOO may be a slightly better as it avoids the bias due to testing the underrepresented class.
But doing repeated and, if adequate for your situation, stratified k-fold CV and taking a look at model stability form this would certainly be better than LOO.