Here is a detailed imaginary example: I am using 5-fold cross-validation to estimate the generalization MSE of my predictive model.
When I hold-out fold number 1, which contains 10 observations, say I obtain:
actual_value predictions squared.residual
1 43.73546 45.57342 3.37807764
2 51.83643 53.40071 2.44694877
3 41.64371 41.79284 0.02223975
4 65.95281 61.97410 15.83008068
5 53.29508 54.53473 1.53673583
6 41.79532 41.68306 0.01260174
7 54.87429 54.56270 0.09708896
8 57.38325 54.44174 8.65245030
9 55.75781 54.80151 0.91450990
10 46.94612 47.78200 0.69870059
Therefore the MSE for the first fold is mean(squared.residual)
. The standard error of this value is sd(squared.residuals)/sqrt(10)
.
So for the first fold I obtain:
MSE SE_MSE
1 3.358943 1.611509
Now imagine that I obtain roughly (or even exactly) the same MSE for each fold. For example:
MSE SE_MSE
1 3.358943 1.611509
2 3.472887 1.680483
3 3.331932 1.614309
4 3.839267 1.537181
5 3.351095 1.630388
The apparent standard error of the MSE is close to zero (or zero in the extreme case where each fold yeilds the same value). Yet, we know the SD of the MSE for each fold and it is not zero at all.
How accurate is my final estimate of the MSE (obtained by averaging the MSE from each fold)?
rnorm
. I'm asking a general question and this dataset is just for illustration purposes. I have a real example here if you want. In the case linked, the folds were obtained by stratified random sampling. $\endgroup$