Comparing two models with statistical testing

Question

I am working on a neural network architecture to tackle a problem and I want to compare my model to another model used in literature. I use k-fold crossvalidation to get a more unbiased accuracy and now I want to compare with statistical tests if one model is better than the other one.

Some more detail: I do a categorical classification for 10 different classes. For each class there are 100 samples, so the dataset has in total 1000 entries. I do 10-fold crossvalidation. Both models reach an accuracy of over 80%.

I did some research (i.e. here on stackexchange) and I had a look into the book "Evaluating Learning Algorithms: A Classification Perspective". I think there are the following tests which suit my needs:

1. The t-test

However, since this test is parametric and assumes a normal distribution, I think this might be a bad fit and I'd better use a non parametric test.

2. The Mann–Whitney U test

As far as I can tell, it has been used in literature and Janez Demsar came to the conclusion, that it is suitable Paper

3. The McNemar's test

I saw this recommendation quite a lot while googling. One problem could be the fact, that there must be at least 30 disagreements (according to the previously mentioned book)

4. The sign test

Seems to be easy to use, however, I did not see it a lot in literature.

Right now I feel a bit lost, since I really do not know, which test to use and which test has advantages to others. Can anyone give me a recommendation or help me to figure out the right choice?

Another fact which might be important is that I only have the average model accuracy after 10-fold crossvalidation from the other paper, but nothing more. I try to rebuild the model, however, if there is a statistics which does not require this, that would, of course, be great, too.

Answer 1

This sounds like a great application for the $k$-fold cross-validated paired $t$ test (Dietterich, 1998). However, you need the accuracies per fold of both your model and the model from the other paper. Having only the mean accuracies over all folds is not sufficient (see alternative below).

The $k$-fold cross-validated paired $t$ test

We have two classification machine learning models $A$ and $B$, that we wish to compare. Under the null hypothesis, both models should have the same test set accuracies $p_A = p_B$. Dietterich (1998) refers to error rates, which is simply $1-$accuracy. The accuracy $p$ is the probability that a randomly drawn sample from the test set will be classified correctly.

To compare both models, we are interested in the difference of the accuracies $p = p_A - p_B$. First, we perform $k$-fold cross-validation by splitting the dataset into $k$ disjoint sets of equal size. We then perform $ k $ experiments, where each set is used as test set and the remaining sets are used for training. Assuming that the $ k $ differences $ p^{(i)} = p_A^{(i)}-p_B^{(i)} $ are drawn independently from a normal distribution, we can perform Student's $ t $ test by computing the statistic

$$ t = \frac{\bar{p}\sqrt{k}}{\sqrt{\frac{\sum_{i=1}^{k} (p^{(i)}-\bar{p})^2}{k-1}}} ~ , $$ where $ \bar{p} = \frac{1}{k} \sum_{i=1}^{k} p^{(i)} $. For $ k=10 $ experiments (10-fold cross-validation), the test statistic has a $t$ distribution with $ k-1=9 $ degrees of freedom. The null hypothesis can be rejected if $ |t| > t_{9,0.975} = 2.262 $ (for a two-sided test with the probability of incorrectly rejecting the null hypothesis of $0.05$).

But what if I only have the mean accuracy over all folds?

In this case, you can still perform the following test. Let $p_A$ and $p_B$ be the mean of the accuracies over $k$ folds, which are assumed to be normally distributed. If the two accuracies are independent, the difference $ p_A - p_B $ is also normally distributed. Assuming that the null hypothesis is true, we can compute the following statistic

$$ z = \frac{p_A - p_B}{\sqrt{2p(1-p)/n}} ~ , $$ where $ p=(p_A + p_B)/2 $ and number of test samples $n$. This test statistic has approximately a standard normal distribution and we can reject the null hypothesis if $ |z| > Z_{0.975} = 1.96 $.

However, this test has several problems as outlined by Dietterich (1998), e.g., $p_A$ and $p_B$ are not independent. Ideally, one should perform the proposed 5x2cv paired $t$ test, if retraining and evaluation of both models being compared is possible.

Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.3325&rep=rep1&type=pdf