Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at hand, in future studies?". To that end, we have one reasonably large sample, and want to apply iterated/repeated cross-validation to compare the accuracy of these classifiers. The goal is really to have a generalizable statement that one classifier might be regarded as more accurate than the other one for this kind of data and research question.

If you apply $K$-fold cross-validation repeated $B$ times, you get $B \times K$ accuracies for each classifier. I can guess, from various answers on SE (e.g., here and here), that using simple tests as t-tests or preferable Wilcoxon tests on those $B \times k$ accuracies should make sense. Is this a possible way, or even the canonical way, to compare the accuracy of two classifiers after such a design? If not, what would be a better way?

Also, how to report extensively the results after an iterated cross-validation? The "global", mean accuracy, is the grand mean over the $B \times K$ accuracies. Should one also report their standard deviation, or any other measure of dispersion? Similarly, is it useful or usual to give a plot (boxplot, stripchart, ...) of those $B \times K$ accuracies for each classifier?


  • 1
    $\begingroup$ The pertinent argument against the t test is not that it is inapplicable when the data are not normally distributed (this argument is false), but the dependence structure introduced by using each observation multiple times. And we actually have two kinds of dependency here, which the Wilcoxon test does not account for. So if at all, it looks like we should be using a more complex mixed model with two different grouping factors. $\endgroup$ May 10, 2021 at 9:05
  • 1
    $\begingroup$ @StephanKolassa do you have any references for a mixed-level approach for analyzing CV data? The problem is not only that there is a fold effect, but that data are dependent across folds, which I think won't be easy to model $\endgroup$
    – rep_ho
    May 10, 2021 at 14:01
  • $\begingroup$ @rep_ho Do you mean something like time series cross validation? $\endgroup$
    – Dave
    May 10, 2021 at 14:02
  • $\begingroup$ @Dave it doesn't matter if it's a time series or any other CV, the CV results are dependent, which is the main problem when trying to get some valid statical inference out of it. Having a 10 fold CV is not the same as testing 10 models on 10 independent samples, which would be relatively easy $\endgroup$
    – rep_ho
    May 10, 2021 at 14:08
  • $\begingroup$ Yes, the multiple dependencies is precisely what I was getting at. Unfortunately, no, I do not have anything I could point you to, or I would have posted it as an answer. Sorry! $\endgroup$ May 10, 2021 at 14:08

1 Answer 1


first, mandatory disclaimer, don't use accuracy as your performance measure unless you really have to, it's quite insensitive especially for model comparison. AUC is slightly better, continuous measures like brier score or log score are much better.

Strictly speaking, you can't really make an inference about other similar datasets based on your one dataset. You can just assume they will be independent samples from the same population. Also, you are not really making statements about how good is the classifier for this problem, but how good is the classifier for this problem, with this CV setup and sample size.

As was mentioned, the problem with CV is that the folds are not independent and what to do with it is kind of an open question. The normality of residuals is not that important and that's easy to solve. If you want a quick and dirty approximate solution that is however better than a naive approach of doing nothing, you can use a correlated t-test where the t-statistic is adjusted for the assumed correlation between folds. See https://arxiv.org/abs/1606.04316 also for some other suggestions on how to deal with this issue (I wouldn't worry too much about their Bayesian preaching). Here is also a newer paper that can give you more pointers https://arxiv.org/pdf/2007.12671.pdf

You can probably also just bootstrap your errors, but I don't have a reference for that, and its easy to make it wrong. If you would care only about a p-value for one model, you can use a permutation test, I don't think that can be used easily also for model comparison https://nmr.mgh.harvard.edu/~fischl/reprints/golland-fischl-ipmi03.pdf (Golland, Fischl 2003; Permutation Tests for Classification: Towards statistical Significance in Image-Based Studies)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.