Comparing the accuracy of binary classifiers using iterated cross-validation

Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at hand, in future studies?". To that end, we have one reasonably large sample, and want to apply iterated/repeated cross-validation to compare the accuracy of these classifiers. The goal is really to have a generalizable statement that one classifier might be regarded as more accurate than the other one for this kind of data and research question.

If you apply $$K$$-fold cross-validation repeated $$B$$ times, you get $$B \times K$$ accuracies for each classifier. I can guess, from various answers on SE (e.g., here and here), that using simple tests as t-tests or preferable Wilcoxon tests on those $$B \times k$$ accuracies should make sense. Is this a possible way, or even the canonical way, to compare the accuracy of two classifiers after such a design? If not, what would be a better way?

Also, how to report extensively the results after an iterated cross-validation? The "global", mean accuracy, is the grand mean over the $$B \times K$$ accuracies. Should one also report their standard deviation, or any other measure of dispersion? Similarly, is it useful or usual to give a plot (boxplot, stripchart, ...) of those $$B \times K$$ accuracies for each classifier?

Thanks!

• The pertinent argument against the t test is not that it is inapplicable when the data are not normally distributed (this argument is false), but the dependence structure introduced by using each observation multiple times. And we actually have two kinds of dependency here, which the Wilcoxon test does not account for. So if at all, it looks like we should be using a more complex mixed model with two different grouping factors. May 10 '21 at 9:05
• @StephanKolassa do you have any references for a mixed-level approach for analyzing CV data? The problem is not only that there is a fold effect, but that data are dependent across folds, which I think won't be easy to model May 10 '21 at 14:01
• @rep_ho Do you mean something like time series cross validation?
– Dave
May 10 '21 at 14:02
• @Dave it doesn't matter if it's a time series or any other CV, the CV results are dependent, which is the main problem when trying to get some valid statical inference out of it. Having a 10 fold CV is not the same as testing 10 models on 10 independent samples, which would be relatively easy May 10 '21 at 14:08
• Yes, the multiple dependencies is precisely what I was getting at. Unfortunately, no, I do not have anything I could point you to, or I would have posted it as an answer. Sorry! May 10 '21 at 14:08