I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a higher test accuracy mean. In best practice, should I use paired t-test, unpaired t-test or other kinds of test? Should I use Welch's test if people are concerned about unequal variances?


The algorithms should be compared on the exact same training/test sets, so a paired test makes sense.

The tricky issue with using a single data set to estimate generalization performance is that data has to be re-used across multiple runs, meaning there's overlap in the training sets (and sometimes test sets, depending on the procedure). This can produce erroneous results because it violates the independence assumption of common statistical tests, and can lead to underestimating the variance.

For comparing the generalization performance of two algorithms on a single dataset, paired t tests with 10-fold cross validation can give an inflated type 1 error (i.e. you'd incorrectly detect a significant difference more often than you should). See this paper:

Dietterich (1998). Approximate statistical tests for comparing supervised classification learning algorithms

Instead, he suggests using the '5x2cv t test' (a paired t test using 5 runs of 2-fold cross validation) or 'McNemar's test' (if computational resources are more limited). Unlike the t test w/ 10-fold cross validation, both of these methods have acceptable type 1 error. But, they have higher type 2 error (meaning a greater probability of failing to detect a true difference).

In this paper:

Nadeau and Bengio (2003). Inference for the generalization error

they propose the 'corrected resampled t test', which adjusts the variance based on overlap between the training/test sets. It has proper type 1 error, and greater statistical power (i.e. lower type 2 error) than the 5x2cv t test and McNemar's test.

In this paper:

Bouckaert and Frank (2004). Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms

they argue that a test should have not only acceptable type 1 error and low type 2 error, but also high replicability (meaning the outcome of the test shouldn't depend strongly on a particular random partitioning of the data). They find that the 5x2cv t test has low replicability. The corrected resampled t test has higher replicability, and they propose a modification that further increases it.

This paper considers the case of comparing multiple algorithms on multiple data sets:

Demsar (2006). Statistical comparisons of classifiers over multiple data sets

  • $\begingroup$ This is an informative answer, thanks! Are there any updates to this now, in 2019? $\endgroup$
    – MD004
    May 24 '19 at 21:53
  • $\begingroup$ +1 Informative answer, indeed. It seems that there is acutally no best way to compare models in a strict stastistically sense. $\endgroup$
    – Funkwecker
    May 8 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.