Timeline for 10-fold Cross-validation vs leave-one-out cross-validation
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 7, 2022 at 17:24 | history | edited | Alexis | CC BY-SA 4.0 |
added 91 characters in body
|
| Dec 19, 2020 at 0:28 | vote | accept | machinery | ||
| Nov 28, 2018 at 9:13 | comment | added | Dikran Marsupial | @RKD314 Sure, these are the ones I typically use doi.org/10.1016/S0031-3203(03)00136-5 doi.org/10.1016/j.neunet.2004.07.002 dx.doi.org/10.1007/s10994-008-5055-9 for SVMs link.springer.com/article/10.1023/A:1012450327387 most books on regression will have the formulae for linear least-squares regression on which the first three are based. HTH | |
| Nov 26, 2018 at 17:09 | comment | added | RKD314 | Could you give some explanation or a link to one on why LOO-CV is computationally inexpensive for the models listed? | |
| Jul 24, 2018 at 10:13 | comment | added | Xavier Bourret Sicotte | Great - @DikranMarsupial have a look at my latest simulation stats.stackexchange.com/questions/280665/… - your thoughts and comments would be welcome ! | |
| Jul 24, 2018 at 8:02 | comment | added | Dikran Marsupial | @XavierBourretSicotte cheers, will give them a read, it is an issue I am interested in (although from a practical point of view, the difference between resampling strategies is often surprisingly small jmlr.org/papers/volume18/16-174/16-174.pdf) | |
| Jul 21, 2018 at 15:05 | comment | added | Xavier Bourret Sicotte | @DikranMarsupial perhaps you have the mathematical background needed for these papers ? math.arizona.edu/~hzhang/math574m/Read/LOOtheory.pdf , theory.stanford.edu/~sergei/papers/ics11-cv.pdf ? | |
| Jul 20, 2018 at 13:15 | comment | added | Dikran Marsupial | very much tempted to perform some simulations with kernel learning methods where I use (virtual) LOOCV quite a bit. | |
| Jul 20, 2018 at 6:21 | comment | added | Dikran Marsupial | cheers @amoeba, for the models I tend to use, I don't think it usually makes a huge difference, perhaps the problem is the variance is larger in "pathological" cases and k-fold is more robust. | |
| Jul 19, 2018 at 20:35 | comment | added | amoeba | Here is another simulation stats.stackexchange.com/a/357749 showing that the variance of CV estimator decreases with the number of folds and LOOCV has the same (or lower) variance as 10-fold. Another simulation linked in my comment above showed another example where variance was decreasing with $k$, and was the lowest for LOOCV. By now I am really curious to see any simulation where the variance would increase with the number of folds. I am also starting to be rather skeptical that it can happen in practice. | |
| May 22, 2017 at 6:10 | comment | added | amoeba | @Dikran This topic (of LOOCV having the largest variance) came up again in a separate and quite interesting question: stats.stackexchange.com/questions/280665, you might want to take a look. | |
| May 20, 2017 at 11:11 | comment | added | usεr11852 | +1 (both the post as well as the latest comment - great paper but not to be blinded followed (as any other paper)). | |
| Mar 1, 2017 at 10:46 | comment | added | Dikran Marsupial | yes, I don't think I agree with it though, as it assumes that the model is stable under the perturbations caused by deleting the test samples, which is only likely to approach being true if you have a very large dataset (i.e. it is only asymptotically true, but if you had that much data, almost any sensible performance evaluation scheme would give you the same result). | |
| Mar 1, 2017 at 10:40 | comment | added | amoeba | Hmm. Doesn't Corollary 2 of this paper say that "The variance of $k$-fold crossvalidation in Proposition 1 does not depend on $k$"? | |
| Mar 1, 2017 at 8:47 | comment | added | Dikran Marsupial | Luntz and Brailovsky is very often cited in papers on SVMs, but it is somewhat ironic that variance tends to be much more of an issue than bias for tuning SVMs and L&B only discuss the unbiasedness, as far as I am aware (my Russian is somewhat limited ;o). I also have a bit of difficulty finding a good explanation. It is easy to show LOOCV can have pathological behaviour, see e.g. example one in Kohavi's paper ai.stanford.edu/~ronnyk/accEst.pdf . | |
| Mar 1, 2017 at 8:43 | comment | added | amoeba | +1 for the obscure 1969 Russian reference! Do you have a good reference for LOOCV having high variance? This is stated in Hastie et al but I am not sure I am 100% convinced by the argument and I haven't seen empirical demonstrations (simulations). | |
| Mar 1, 2017 at 8:39 | history | answered | Dikran Marsupial | CC BY-SA 3.0 |