Timeline for Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of "stability"?
Current License: CC BY-SA 4.0
31 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 13, 2023 at 17:02 | answer | added | civilstat | timeline score: 4 | |
| Oct 10, 2018 at 23:30 | answer | added | user39663 | timeline score: 3 | |
| S Jul 27, 2018 at 7:56 | history | bounty ended | amoeba | ||
| S Jul 27, 2018 at 7:56 | history | notice removed | amoeba | ||
| S Jul 24, 2018 at 20:34 | history | bounty started | amoeba | ||
| S Jul 24, 2018 at 20:34 | history | notice added | amoeba | Reward existing answer | |
| Jul 24, 2018 at 19:31 | history | edited | Jake Westfall | CC BY-SA 4.0 |
changed question title to better reflect the main focus of the question
|
| Jul 24, 2018 at 13:35 | vote | accept | Jake Westfall | ||
| Jul 21, 2018 at 6:55 | answer | added | Xavier Bourret Sicotte | timeline score: 17 | |
| Jul 18, 2018 at 11:44 | comment | added | amoeba | Another simulation supporting your conclusions that the variance decreases with $K$: stats.stackexchange.com/a/357749/28666. | |
| Dec 1, 2017 at 14:53 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Oct 3, 2017 at 22:23 | answer | added | user32398 | timeline score: 1 | |
| S May 30, 2017 at 7:11 | history | bounty ended | CommunityBot | ||
| S May 30, 2017 at 7:11 | history | notice removed | CommunityBot | ||
| May 26, 2017 at 17:11 | comment | added | amoeba | I don't think so, @Jake. What I wrote invalidates your "counter-intuition", but the main "intuition" (about models from different folds being highly dependent) still can hold. | |
| May 26, 2017 at 16:08 | comment | added | Jake Westfall | @amoeba Yes I see your point. Which seems to intuitively support that "The variance of K fold cross validation [...] does not depend on K" as one of the papers concludes. | |
| May 26, 2017 at 12:37 | comment | added | amoeba | I don't think your "counter-intuition" is a valid argument, Jake. The mean error of K-fold CV is indeed the mean over K folds, but the error obtained in each fold is in turn the mean over N/K samples in the test set. Overall, the mean error of K-fold CV is effectively the mean over N data points, and this does not depend on K. | |
| May 25, 2017 at 21:32 | comment | added | Luca Citi | Different but related issue. In classification LOO may underestimate the accuracy of the classifier because each fold's predictor is trained on data where the prior probability of the class in the test fold is less than in the whole dataset. I have seen LOO returning less than 50% accuracy on a binary classification task. Stratified LOO (i.e. leave-one-per-class-out) helps mitigate this problem. | |
| S May 22, 2017 at 6:08 | history | bounty started | amoeba | ||
| S May 22, 2017 at 6:08 | history | notice added | amoeba | Authoritative reference needed | |
| May 20, 2017 at 15:08 | comment | added | Jake Westfall | @ŁukaszGrad Wonderful, I didn't see that there is a wikipedia article about stability! And it provides a paper reference backing up the idea that linear regression (with or without regularization) is an example of a stable algorithm. I do still hope that someone can provide a nice illustration of the idea, unpack the formidable technical definition given in the "Definitions" section, etc... | |
| May 20, 2017 at 15:03 | history | edited | Jake Westfall | CC BY-SA 3.0 |
Added two more examples of conventional explanation
|
| May 20, 2017 at 14:43 | comment | added | Jake Westfall | @amoeba "mean" is, as you guessed, the mean CV estimate across the 10000 datasets. I agree it would be more meaningful to represent this as "bias" (mean - true generalization error), the problem is just that I'm not sure how to work out the true error from the parameters of the simulation :). I'm tempted to say that it's 1, because the fitted regression models are unbiased and $\sigma=1$, but I'm not sure that's actually correct. Also, yes, I believe true generalization error ("prediction error" may have been a better term) is an expected value over all possible datasets of size $N=50$ | |
| May 20, 2017 at 11:23 | comment | added | usεr11852 | If stability means "that an algorithm produces similar results on training set with $N$ and $N−1$ examples." then for reasonably large samples most methods are stable. Similarly in such cases LOOCV can become unattainable and we need to use GCV or something similar. In general I would never suggest the use of $K$-fold cross-validation on it's own. As a rule of thumb I would suggest using repeated $K$-fold cross-validation unless evidence for the opposite arise. (Probably something like 20 times 5-fold - for papers probably 100$\times$ 5-fold - eg. here) | |
| May 20, 2017 at 8:33 | comment | added | Łukasz Grad | I may not make any sense but on a longer thought - lets say we have some unstable algorithm that is very sensitive to outliers and moreover one example in dataset is such an outlier that heavily impacts our model. Then, informally, we weigh this impact with $\frac{k-1}{k}$, so in LOOCV it is the highest | |
| May 20, 2017 at 8:10 | history | tweeted | twitter.com/StackStats/status/865842253637578754 | ||
| May 20, 2017 at 8:01 | comment | added | amoeba | Apart from that, I have recently talked about it with @DikranMarsupial (who is probably one of our main experts on cross-validation here on CV) here in the comments - he suggested to read Kohavi's 1995 paper. Dikran was also talking about stability. Unfortunately, I did not follow it up since then. | |
| May 20, 2017 at 8:00 | comment | added | Łukasz Grad | +1. After short glance at en.wikipedia.org/wiki/… it seems that in this context stability means that an algorithm produces similar results on training set with $N$ and $N-1$ examples. Where similar means difference w.r.t. some loss function bounded by some low value | |
| May 20, 2017 at 7:54 | comment | added | amoeba | +1. What exactly is "mean" in your simulation results? Mean CV estimate of the generalization error (mean across 10000 datasets)? But what should we compare it to? It would be more meaningful to show the bias, i.e. root-mean-square-deviation from the true generalization error. Also, what is "true generalization error" in this case? True generalization error of the estimate on a given N=100 dataset? Or expected value of the true generalization error (expected value over all N=100 datasets)? Or something else? | |
| May 20, 2017 at 1:16 | history | edited | Jake Westfall | CC BY-SA 3.0 |
added 10 characters in body
|
| May 20, 2017 at 1:11 | history | asked | Jake Westfall | CC BY-SA 3.0 |