Timeline for Why does k-fold cross validation generate an MSE estimator that has higher bias, but lower variance then leave-one-out cross-validation?
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 11, 2019 at 13:22 | answer | added | cbeleites | timeline score: 1 | |
| Sep 7, 2018 at 7:05 | comment | added | amoeba | @XavierBourretSicotte Your answer, with which I agree, has my +1. Frankly, I am puzzled why DenizenOfTheNorth does not agree with us. The premise of this question ("the denominator is larger as well because there are essentially n estimates (greater than k estimates as in the k-fold case)") is simply a misunderstanding. | |
| Sep 4, 2018 at 22:18 | comment | added | Xavier Bourret Sicotte | I don't understand why you and Amoeba don't agree - there is no K in the formula so how could K have an impact on the bias or variance ? The impact comes from the correlation / similarity / stability of the training sets and the algorithm | |
| Sep 4, 2018 at 21:15 | comment | added | denizen of the north | @XavierBourretSicotte thanks for pointing it out. That is exactly the point i am trying to make. If it is the summation sign or the formula to expand the variance of the summation of random variables that got people caught up, there is really no point in proceeding the argument... | |
| Sep 4, 2018 at 10:12 | comment | added | Xavier Bourret Sicotte | @amoeba I have edited my answer: Varying K does not have a direct, algebraically straightforward impact on the variance of the CV estimator and These two effects are influenced by the value of K which explains why different datasets and models will lead to different behaviours - This question and answer clarifies a lot of confusion on the topic I think and should be linked to more often | |
| Jul 23, 2018 at 14:09 | comment | added | amoeba | No. I strongly disagree with this assessment. | |
| Jul 23, 2018 at 14:05 | comment | added | denizen of the north | @amoeba I see your point. It probably can be used as a heuristic, but definitely not correct in a rigorous sense. The mean MSE is different for each fold. What you are proposing is essentially a global MSE estimate using the global mean. The whole point of k-fold MSE estimation is to have k estimates rather than one global estimate. | |
| Jul 23, 2018 at 13:58 | comment | added | amoeba | What you are missing is that an estimate of MSE in each fold is itself an average over $n/k$ estimates. So you have an average of $k$ numbers that are averages of $n/k$ numbers. That's the same as an average of $n$ numbers. Does it make sense? | |
| Jul 23, 2018 at 13:32 | comment | added | denizen of the north | it is been a while, but lets talk it through. So for k-fold cross-validation, we have one estimate of MSE in each folder, so we have k estimates. The aggregate estimate we have is $\Sigma \hat{MSE_i}/k$. If there are n folds, then the n is the denominator. | |
| Jul 23, 2018 at 8:28 | comment | added | amoeba | $k$-fold CV with any value of $k$ produces an error for each of the $n$ observations. So MSE estimate always has the denominator $n$. This denominator does not change between LOOCV and e.g. 10-fold CV. This is your main confusion here. | |
| Jul 23, 2018 at 8:27 | history | edited | amoeba |
edited tags
|
|
| Jul 23, 2018 at 7:12 | answer | added | Xavier Bourret Sicotte | timeline score: 6 | |
| Jan 26, 2018 at 3:17 | history | asked | denizen of the north | CC BY-SA 3.0 |