I am working on a neural network architecture to tackle a problem and I want to compare my model to another model used in literature. I use k-fold crossvalidation to get a more un-biased accuracy and now I want to compare with statistical tests if one model is better than the other one.

Some more detail: I do a categorical classification for 10 different classes. For each class there are 100 samples, so the dataset has in total 1000 entries. I do 10-fold crossvalidation. Both models reach an accuracy of over 80%.

I did some research (i.e. here on stackexchange) and I had a look into the book "Evaluating Learning Algorithms: A Classification Perspective". I think there are the following tests which suit my needs:

1. The t-test

However, since this test is parametric and assumes a normal distribution, I think this might be a bad fit and I'd better use a non parametric test.

2. The Mann–Whitney U test

As far as I can tell, it has been used in literature and Janez Demsar came to the conclusion, that it is suitable Paper

3. The McNemar's test

I saw this recommendation quite a lot while googling. One problem could be the fact, that there must be at least 30 disagreements (according to the previously mentioned book)

4. The sign test

Seems to be easy to use, however, I did not see it a lot in literature.

Right now I feel a bit lost, since I really do not know, which test to use and which test has advantages to others. Can anyone give me a recommendation or help me to figure out the right choice?

Another fact which might be important is that I only have the average model accuracy after 10-fold crossvalidation from the other paper, but nothing more. I try to rebuild the model, however, if there is a statistics which does not require this, that would of course be great, too.

Best heyhey

I am working on a neural network architecture to tackle a problem and I want to compare my model to another model used in literature. I use k-fold crossvalidation to get a more unbiased accuracy and now I want to compare with statistical tests if one model is better than the other one.

Some more detail: I do a categorical classification for 10 different classes. For each class there are 100 samples, so the dataset has in total 1000 entries. I do 10-fold crossvalidation. Both models reach an accuracy of over 80%.

I did some research (i.e. here on stackexchange) and I had a look into the book "Evaluating Learning Algorithms: A Classification Perspective". I think there are the following tests which suit my needs:

1. The t-test

However, since this test is parametric and assumes a normal distribution, I think this might be a bad fit and I'd better use a non parametric test.

2. The Mann–Whitney U test

As far as I can tell, it has been used in literature and Janez Demsar came to the conclusion, that it is suitable Paper

3. The McNemar's test

I saw this recommendation quite a lot while googling. One problem could be the fact, that there must be at least 30 disagreements (according to the previously mentioned book)

4. The sign test

Seems to be easy to use, however, I did not see it a lot in literature.

Right now I feel a bit lost, since I really do not know, which test to use and which test has advantages to others. Can anyone give me a recommendation or help me to figure out the right choice?

Another fact which might be important is that I only have the average model accuracy after 10-fold crossvalidation from the other paper, but nothing more. I try to rebuild the model, however, if there is a statistics which does not require this, that would, of course, be great, too.