# cross-validation: what is the standard deviation if the same value is obtained for each fold?

Here is a detailed imaginary example: I am using 5-fold cross-validation to estimate the generalization MSE of my predictive model.

When I hold-out fold number 1, which contains 10 observations, say I obtain:

   actual_value predictions squared.residual
1      43.73546    45.57342       3.37807764
2      51.83643    53.40071       2.44694877
3      41.64371    41.79284       0.02223975
4      65.95281    61.97410      15.83008068
5      53.29508    54.53473       1.53673583
6      41.79532    41.68306       0.01260174
7      54.87429    54.56270       0.09708896
8      57.38325    54.44174       8.65245030
9      55.75781    54.80151       0.91450990
10     46.94612    47.78200       0.69870059


Therefore the MSE for the first fold is mean(squared.residual). The standard error of this value is sd(squared.residuals)/sqrt(10).

So for the first fold I obtain:

       MSE   SE_MSE
1 3.358943 1.611509


Now imagine that I obtain roughly (or even exactly) the same MSE for each fold. For example:

       MSE   SE_MSE
1 3.358943 1.611509
2 3.472887 1.680483
3 3.331932 1.614309
4 3.839267 1.537181
5 3.351095 1.630388


The apparent standard error of the MSE is close to zero (or zero in the extreme case where each fold yeilds the same value). Yet, we know the SD of the MSE for each fold and it is not zero at all.

How accurate is my final estimate of the MSE (obtained by averaging the MSE from each fold)?

• How were the splits obtained? Randomly? Aug 29, 2015 at 11:51
• This is an imaginary example generated with some calls to rnorm. I'm asking a general question and this dataset is just for illustration purposes. I have a real example here if you want. In the case linked, the folds were obtained by stratified random sampling.
– asac
Aug 29, 2015 at 13:42

Dealing with regression can be confusing because there are 2 SD. The whole point of the cross validation is to give you an estimate of the future behavior of the regressor. In this case you have 5 estimations of the regressor on future data, one for each fold.

What do you want to know about the regressor on future data:

1) what is the expected error (MSE) on future data - that is the mean of the 5 CV MSEs

2) what is the expected SD of the errors on future data - that is the mean of each CV SD!

You may also want to evaluate how certain you are of those estimates:

3) what is the variance (or SD) of the estimate of future MSE - that is the variance (SD) of the 5 CV MSEs -- this is the one that is low in your example - so you know that your estimate of the future MSE is pretty tight. (Well maybe - there is a paper that shows that the SD of a CV measure - in this case the MSE - is not a good estimate of the true SD - but let us leave this for later)

4) what is the variance (or SD) of the estimate of future SD of the MSE - that is the variance (SD) of the 5 CV SD

So going back to your question

The apparent standard error of the MSE is close to zero (or zero in the extreme case where each fold yeilds the same value). (That is the SD for the estimate of the MSE and not the estimate of the SD for future data) Yet, we know the SD of the MSE for each fold and it is not zero at all. (That is the estimate of future SD)

• Ooooh, of course! Thanks for the explanation! I do have a follow-up question though: you say the SD of the 5 CV MSEs gives (more or less) the variance of my estimate of the MSE, and I agree. But this estimate is the mean of the 5 CV MSEs. And the standard error of this mean is therefore SD(CV MSEs)/sqrt(5). So what does this last quantity represent? Why is it different than the SD of the 5 CV MSEs? This is basically my question here which I think you could answer easily :)
– asac
Aug 29, 2015 at 14:43
• I believe that explanation 2) is incorrect. The expected SD of the errors on future data is not the mean of the 5 CV SDs. The individual SDs would need to be squared, then averaged across folds, and rooted - alternatively, just rooting the multi-fold MSE from 1). Jul 12, 2016 at 2:30

Of course, you may have found a random split that just accidentally leads to the same results for each split. This possibilty can easily be excluded by running a few more iterations/repetitions of the $k$-fold cross validation.

If on the other hand, your splits were obtained in a non-random fashion such as venetian blinds and you have a situation where the prediction acurracy is a function of the value, this would lead to such a situation where the variance of the MSE between the splits is much less than the variance of the squared errors within the splits.