I am working on a neural network architecture to tackle a problem and I want to compare my model to another model used in literature. I use k-fold crossvalidation to get a more unbiased accuracy and now I want to compare with statistical tests if one model is better than the other one.
Some more detail: I do a categorical classification for 10 different classes. For each class there are 100 samples, so the dataset has in total 1000 entries. I do 10-fold crossvalidation. Both models reach an accuracy of over 80%.
I did some research (i.e. here on stackexchange) and I had a look into the book "Evaluating Learning Algorithms: A Classification Perspective". I think there are the following tests which suit my needs:
- The t-test
However, since this test is parametric and assumes a normal distribution, I think this might be a bad fit and I'd better use a non parametric test.
- The Mann–Whitney U test
As far as I can tell, it has been used in literature and Janez Demsar came to the conclusion, that it is suitable Paper
- The McNemar's test
I saw this recommendation quite a lot while googling. One problem could be the fact, that there must be at least 30 disagreements (according to the previously mentioned book)
- The sign test
Seems to be easy to use, however, I did not see it a lot in literature.
Right now I feel a bit lost, since I really do not know, which test to use and which test has advantages to others. Can anyone give me a recommendation or help me to figure out the right choice?
Another fact which might be important is that I only have the average model accuracy after 10-fold crossvalidation from the other paper, but nothing more. I try to rebuild the model, however, if there is a statistics which does not require this, that would, of course, be great, too.
