9
$\begingroup$

Are standard deviation estimates calculated via:

$ s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2}. $

(http://en.wikipedia.org/wiki/Standard_deviation#Sample_standard_deviation)

for prediction accuracies sampled from 10-fold cross validation? I'm concerned that the prediction accuracy calculated between each fold are dependent because of the substantial overlap between training sets (although the prediction sets are independent). Any resources that discuss this would be very helpful.

$\endgroup$
1
$\begingroup$

I'm concerned that the prediction accuracy calculated between each fold are dependent because of the substantial overlap between training sets (although the prediction sets are independent).

IMHO the overlap between the training sets does not need to be a big concern here. That is, it is of course important to check whether the models are stable. Stable implies that the cross validation surrogate models' predictions are equivalent (i.e. an independent case would get the same prediction by all those models), and in fact cross validaton usually claims equivalence not only between the surrogate models but also to the model trained on all cases. So this dependence is rather a consequence of what we want to have.

This applies for the typical question: if I train a model on these data, what are the prediction intervals? If the question is instead, if we train a model on $n$ cases of this population, what are the prediction intervals?, we cannot answer it because that overlap in the training sets means we underestimate variance by an unknown amount.

What are the consequences compared to testing with an independent test set?

  • Cross validation estimates may have higher variance than testing the final model with an independent test set of the same size, because in addition to the variance due to test cases we face variance due to instability of the surrogate models.
  • However, if the models are stable, this variance is small/negligible. Moreover this type of stability can be measured.

  • What can not be measured is how representative the whole data set is compared to the population it was drawn from. This includes part of the bias of the final model (however, also a small independent test set may have a bias) and it means that the corresponding variance cannot be estimated by cross validation.

  • In application practice (performance of model trained on these data), the prediction interval calculation would face issues that IMHO are more important than what part of variance cross validation cannot detect: e.g.

    • cross validation cannot test performance for cases that are independent in time (predictions are usually needed for cases that are measured in the future)
    • the data may contain unknown clusters, and out-of-cluster performance may be important. Clustered data is in priciple something you can account for in cross validation, but you need to know about the clustering.

    These are more than just a cross validation vs. independent test set thing: basically you'd need to sit down and design a validation study, otherwise there's a high risk that the "independent" test set is not all that independent. Once that is done, one can think about which factors are likely to be of practical importance and which can be neglected. You may arrive at the conclusion that after thorough consideration, cross valiation is good enough and the sensible thing to do because the independent validation would be far too expensive compared to the possible information gain.

All things put together, I'd use the usual formula for the standard deviation, call it $s_{CV}$ in analogy to $RMSE_{CV}$ and report in detail how the testing was done.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.