Timeline for Choice of K in K-fold cross-validation
Current License: CC BY-SA 3.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 25, 2022 at 19:14 | comment | added | cbeleites | ... have surprisingly large pessimistic bias, but (randomly) not all of the train/test splits will have the problem, so it is less pessimistic. Note that LOO is mostly preferred over k-fold with extremely small sample sizes, where this problem is more pronounced. So for k -> n the part of the pessimistic bias that is due to the surrogate training sets being systematically smaller than the whole data set decreases - but this is only one of the reasons that can lead to pessimistic bias. | |
| Nov 25, 2022 at 19:12 | comment | added | cbeleites | @Maha, here's a textbook example: consider a very simple binary classifier that always predicts the majority class of its training data, and a balanced (relative frequencies 1 : 1) data set. Now, with k-fold, it is possible to set up stratified resampling, i.e. the surrogate training and test sets preserve the 1 : 1 relative frequencies. This is not possible with LOO, the left-out case will always belong to the minority class in the surrogate training set, and in consequence, it will estimate accuracy to be 0 - the worst pessimistic bias possible. k-fold CV without stratification will also... | |
| Nov 25, 2022 at 15:55 | comment | added | Mahesha999 | Why you say "usually large $k$ mean less (pessimistic) bias. (Some exceptions are known particularly for $k=n$, i.e. leave-one-out)."? There are many posts that discuss why leave one out has lower bias. For exanple, This says "low bias is given because the training set is very large - almost identical with the entire dataset (as only one data sample is left out for testing). So, for one particular dataset we can expect a very good estimation." | |
| S Jul 20, 2015 at 1:31 | history | suggested | smci | CC BY-SA 3.0 |
correct typo
|
| Jul 20, 2015 at 0:52 | review | Suggested edits | |||
| S Jul 20, 2015 at 1:31 | |||||
| Sep 14, 2014 at 14:41 | comment | added | cbeleites | @jpcgandre: (other $k$) sure - why not: that's part of what I mean with $k$ being a somewhat arbitrary choice. | |
| Sep 14, 2014 at 14:24 | comment | added | cbeleites | I'd say that if we think 80x 5-fold vs. 40x 10-fold could make a serious difference, then we're not in a situation where we should do any grid search optimization (as discussed in the Cawley paper): too few cases anyways. And yes, this is because of the large variance - but variance due to a source we cannot tackle by our cross validation/out of bootstrap setup. | |
| Sep 14, 2014 at 14:22 | comment | added | cbeleites | @jpcgandre: At least for classification errors such as sensitivity, specificity etc., uncertainty due to total number of tested cases can be calculated. While it is true that this is only part of the total variance, at least in the situations I encounter in my work, this uncertainty is often so large that even a rough guesstimate is enough to make clear that conclusions are severely limited. And this limitation stays, it won't go away by using 50x 8-folds or 80x 5-folds instead of 40x 10-fold cross validation. | |
| Sep 13, 2014 at 8:23 | comment | added | jpcgandre |
Also, returning to my original point http://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf say that: while unbiasedness is reassuring, as it means that the form of the model selection criterion is correct on average , the variance of the criterion is also vitally important as it is this that ensures that the minimum of the selec tion criterion evaluated on a particular sample will provide good generalisation.
|
|
| Sep 12, 2014 at 11:03 | comment | added | jpcgandre | @ceibeles: However, the statistical uncertainty due to limited sample size is in practice unknown and hard do reduce and one usually focuses in sources more amenable such as model uncertainties. In this context, for small samples and for the same appropriate i number (i x k-folds) if extreme values are present in the data (but which should not discarded as a pre-condition in this case) it may be preferable to choose a k value smaller than 10. | |
| Sep 12, 2014 at 10:34 | comment | added | jpcgandre | @cbeleites: I agree with you. Variance due to limited sample size usually dominates over model uncertainty. | |
| Sep 12, 2014 at 10:25 | comment | added | cbeleites | @jpcgandre: I'd say it is preferrable to have low total error. Keep in mind that the variance reduction by iterations is limited to variance that is caused by model instability, and that there's no way to get around the fact that with whatever number of folds, in the end you only have $n$ independent cases. Assuming you do enough iterations, the variance is IMHO practically independent of the number of folds (I tend to compare same total number of surrogate models e.g. 100x 10-fold vs. 200x5-fold), but rather dominated by variance due to the available sample size. | |
| Sep 12, 2014 at 3:13 | comment | added | jpcgandre | @cbeleites: Isn't it preferable to have bias and low variance than low bias and high variance? If so, then for small datasets maybe repeating a large number of times a small k (5) cross validation is best | |
| Nov 28, 2012 at 16:44 | vote | accept | Charles Menguy | ||
| May 6, 2012 at 13:46 | comment | added | cbeleites | @steffen, looking into such model comparisons is next on my list of things that I'd really like to look into. I personally have a lot of doubts about these data-driven optimizations. I've experienced extreme overfitting in the inner CV in some situations - and while I think the outer CV gives a sensible estimate of the resulting model's performance. For the moment I came to the conclusion that I get about as good models by deciding model parameters by my knowledge on the subject and my general experience with the models. | |
| May 4, 2012 at 15:45 | comment | added | steffen | I see. I agree that the approach is valid to estimate the stability of the surrogate. What I had back in mind was the follow-up-statistical test to decide whether one model outperforms another one. Repeating a cv way too often increases the chance of an alpha error unpredictably. So I was confusing the inner with the outer validation (as dikran has put it here). | |
| May 4, 2012 at 12:35 | comment | added | cbeleites | @steffen, isn't that what ogrisel already pointed out: that the (surrogate) models are not really independent? I completely agree that this is the case. Actually, I try to take this into account by interpreting the results in terms of stability of the (surrogate) models wrt. exchanging "a few" samples (which I didn't want to elaborate here - but see e.g. stats.stackexchange.com/a/26548/4598). And I do not calculate standard error but rather report e.g. median and $5^{th}$ to $95^{th}$ percentile of the observed errors over the iterations. I'll post a separate question about that. | |
| May 4, 2012 at 11:31 | comment | added | steffen | (+1) for the elaboration, but (-1) for the repetition counts of the CV. It is true, that the risk of creating exact duplicats (looking at the ids of the observations) is small (given enough data etc.), but the risk of creating pattern/ data structure duplicates is very high. I would not repeat a CV more than 10 times, no matter what k is ... just to avoid underestimation of the variance. | |
| May 4, 2012 at 6:04 | history | answered | cbeleites | CC BY-SA 3.0 |