I use logistic regression for a binary prediction problem and use LOOCV to estimate the models' prediction accuracies for different predictor sets. A reviewer wants me to use k-fold CV instead. Does this make sense? I would think LOOCV would dominate k-fold CV, and should be preferred unless it it too computationally demanding. Isn't it better to estimate the sub-models on larger samples?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ Did you see the various previous discussion on this, e.g. stats.stackexchange.com/questions/154830/… or stats.stackexchange.com/questions/90902/… ? $\endgroup$– BjörnCommented Sep 10, 2021 at 7:55
-
2$\begingroup$ It is possible that k-fold is better than LOO. See e.g. James et al. textbook "Introduction to Statistical Learning", they have a discussion about that. $\endgroup$– Richard HardyCommented Sep 10, 2021 at 9:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
I would think LOOCV would dominate k-fold CV, and should be preferred unless it it too computationally demanding.
Actually, no.
Isn't it better to estimate the sub-models on larger samples?
We expect the sub-models to be more similar to the model (i.e. the model trained on the full data set) in many cases, leading to lower bias, yes.
OTOH, LOO will always (systematically) test with a case that is from a class that is underrepresented compared to the training set of the model. For training algorithms that estimate something along the lines of relative class frequency from the training data, this can lead to a surprisingly large pessimistic bias in the LOO estimate.
With a cross validation scheme that leaves out more cases, the problem is less severe, and can even be deliberately counteracted by stratified k-fold CV. Whether this is a sensible choice or not of course depends on the situation at hand.
In addition, k-fold CV can be repeated (aka iterated) with new random partitions.
Doing this has two advantages: You can average out the effect of model instabiliy (leading to lower variance uncertainty on the estimate), and (IMHO even more importantly) you can separate the effects of having a finite test sample size from the effects of model instability. This is not possible with LOO, since LOO has perfect correlation between left out sample and surrogate model.
So, doing a standard 10-fold CV instead of LOO may be a slightly better as it avoids the bias due to testing the underrepresented class.
But doing repeated and, if adequate for your situation, stratified k-fold CV and taking a look at model stability form this would certainly be better than LOO.
answered Sep 11, 2021 at 16:35
-
$\begingroup$ "OTOH, LOO will always (systematically) test with a case that is from a class that is underrepresented compared to the training set of the model." Can you not turn this argument on its head? Isn't it the case that classes of the elements of the data set are likely to be overrepresented compared to the underlying distribution unless some stratification procedure was used to generate it? By that argument; isn't the pessimism of the LOOCV warranted? $\endgroup$ Commented Jan 2, 2023 at 11:53
-
$\begingroup$ @FaultyBagnose: the LOO training set always has a composition that compared to the whole data underrepresents the class that is to be tested and overrepresents the other classes. A correct sampling procedure to generate the whole data set from the underlying distribution will have variance around the true relative frequencies of the classes, so in practice some classes will be overrepresented, and others underrepresented. But the combination of LOO with certain algorithms will turn this variance into a bias. This is not just "unstable bad", it is "additionally bad"... $\endgroup$ Commented Jan 2, 2023 at 23:34
-
$\begingroup$ ... With that I mean that the LOO estimate isn't even a good estimate of the performance of the model trained from the whole data set - whatever deviation that model has from the true class boundaries. $\endgroup$ Commented Jan 2, 2023 at 23:36
-
$\begingroup$ This I don't understand; Assume your dataset is drawn iid from an underlying distribution, and you pick a random element from your dataset. Then the class of that element will tend to be overrepresented in your dataset, compared to the underlying distribution, right? This is trivial if your dataset has a single element, but still true for larger sets. $\endgroup$ Commented Jan 4, 2023 at 12:43
-
$\begingroup$ @FaultyBagnose: But LOO would pool the test results for all cases in your data set, and at that level there is no difference any more from the "what samples are tested" perspective compared to k-fold. $\endgroup$ Commented Jan 15, 2023 at 22:25