I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a higher test accuracy mean. In best practice, should I use paired t-test, unpaired t-test or other kinds of test? Should I use Welch's test if people are concerned about unequal variances?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
The algorithms should be compared on the exact same training/test sets, so a paired test makes sense.
The tricky issue with using a single data set to estimate generalization performance is that data has to be re-used across multiple runs, meaning there's overlap in the training sets (and sometimes test sets, depending on the procedure). This can produce erroneous results because it violates the independence assumption of common statistical tests, and can lead to underestimating the variance.
For comparing the generalization performance of two algorithms on a single dataset, paired t tests with 10-fold cross validation can give an inflated type 1 error (i.e. you'd incorrectly detect a significant difference more often than you should). See this paper:
Dietterich (1998). Approximate statistical tests for comparing supervised classification learning algorithms
Instead, he suggests using the '5x2cv t test' (a paired t test using 5 runs of 2-fold cross validation) or 'McNemar's test' (if computational resources are more limited). Unlike the t test w/ 10-fold cross validation, both of these methods have acceptable type 1 error. But, they have higher type 2 error (meaning a greater probability of failing to detect a true difference).
In this paper:
Nadeau and Bengio (2003). Inference for the generalization error
they propose the 'corrected resampled t test', which adjusts the variance based on overlap between the training/test sets. It has proper type 1 error, and greater statistical power (i.e. lower type 2 error) than the 5x2cv t test and McNemar's test.
In this paper:
Bouckaert and Frank (2004). Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms
they argue that a test should have not only acceptable type 1 error and low type 2 error, but also high replicability (meaning the outcome of the test shouldn't depend strongly on a particular random partitioning of the data). They find that the 5x2cv t test has low replicability. The corrected resampled t test has higher replicability, and they propose a modification that further increases it.
This paper considers the case of comparing multiple algorithms on multiple data sets:
Demsar (2006). Statistical comparisons of classifiers over multiple data sets
-
$\begingroup$ This is an informative answer, thanks! Are there any updates to this now, in 2019? $\endgroup$– MD004May 24, 2019 at 21:53
-
$\begingroup$ +1 Informative answer, indeed. It seems that there is acutally no best way to compare models in a strict stastistically sense. $\endgroup$ May 8, 2021 at 15:19
$\begingroup$
$\endgroup$
A follow up on @user20160 comment, and based on Bouckaert and Frank (2004), you can use 10 times repeated 10 fold cross validation while taking into account two points:
Reduce Type 1 error by multiplying the variance with a constant coefficient based onNadeau and Bengi correction.
Tackle the interdependence issue between the fold by reducing the degrees of freedom from 99 to 10.
I think this is one of the compromised solutions I found out there!
