Comparing the accuracy of binary classifiers using iterated cross-validation

Question

Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at hand, in future studies?". To that end, we have one reasonably large sample, and want to apply iterated/repeated cross-validation to compare the accuracy of these classifiers. The goal is really to have a generalizable statement that one classifier might be regarded as more accurate than the other one for this kind of data and research question.

If you apply $K$-fold cross-validation repeated $B$ times, you get $B \times K$ accuracies for each classifier. I can guess, from various answers on SE (e.g., here and here), that using simple tests as t-tests or preferable Wilcoxon tests on those $B \times k$ accuracies should make sense. Is this a possible way, or even the canonical way, to compare the accuracy of two classifiers after such a design? If not, what would be a better way?

Also, how to report extensively the results after an iterated cross-validation? The "global", mean accuracy, is the grand mean over the $B \times K$ accuracies. Should one also report their standard deviation, or any other measure of dispersion? Similarly, is it useful or usual to give a plot (boxplot, stripchart, ...) of those $B \times K$ accuracies for each classifier?

Thanks!

Answer 1

first, mandatory disclaimer, don't use accuracy as your performance measure unless you really have to, it's quite insensitive especially for model comparison. AUC is slightly better, continuous measures like brier score or log score are much better.

Strictly speaking, you can't really make an inference about other similar datasets based on your one dataset. You can just assume they will be independent samples from the same population. Also, you are not really making statements about how good is the classifier for this problem, but how good is the classifier for this problem, with this CV setup and sample size.

As was mentioned, the problem with CV is that the folds are not independent and what to do with it is kind of an open question. The normality of residuals is not that important and that's easy to solve. If you want a quick and dirty approximate solution that is however better than a naive approach of doing nothing, you can use a correlated t-test where the t-statistic is adjusted for the assumed correlation between folds. See https://arxiv.org/abs/1606.04316 also for some other suggestions on how to deal with this issue (I wouldn't worry too much about their Bayesian preaching). Here is also a newer paper that can give you more pointers https://arxiv.org/pdf/2007.12671.pdf

You can probably also just bootstrap your errors, but I don't have a reference for that, and its easy to make it wrong. If you would care only about a p-value for one model, you can use a permutation test, I don't think that can be used easily also for model comparison https://nmr.mgh.harvard.edu/~fischl/reprints/golland-fischl-ipmi03.pdf (Golland, Fischl 2003; Permutation Tests for Classification: Towards statistical Significance in Image-Based Studies)