I want to compare the accuracy of two machine learning (ML) methods, each of which have two classifiers within them. I have six subjects to test these two ML methods against. The same subjects, data, and everything in between is the same for both ML methods.

I've run the tests and the results are presented in this form, where each subject/classifier row is a percentage of accuracy:

ML method 1            ML method 2
-------------------    -------------------
sub 1, classifier 1    sub 1, classifier 1
sub 2, classifier 1    sub 2, classifier 1
...                    ...
sub 6, classifier 1    sub 6, classifier 1
sub 1, classifier 2    sub 1, classifier 2
sub 2, classifier 2    sub 2, classifier 2
...                    ...
sub 6, classifier 2    sub 6, classifier 2 

To get the results subject-wise, I take the average of each classifier per subject (mean(subx_class1, subx_class2), where x is subject 1 to 6), which gives me:

ML method 1            ML method 2
-------------------    -------------------
sub 1                  sub 1
sub 2                  sub 2
sub 3                  sub 3
sub 4                  sub 4
sub 5                  sub 5      
sub 6                  sub 6

My questions is, what would be the best statistical method to use in this example to determine if the two ML methods produce different accuracies which are statistically significant? I don't have a great deal of knowledge in this area, and I've mostly dealt with paired t-tests before, which I don't think I can do in this example due to the data not being subject specific.

Thanks very much for any help.


1 Answer 1


This would depend on your classification problem. If your classification is binary (ie has cancer/doesn't have cancer), you can use McNemar's test directly. If your problem has multiple classes (dog/cat/lizard) you could convert to correct/incorrect and use McNemar's test.

Especially for your sample size, if it is not obvious upon inspection that one method is better, a statistical test will not tell you much about the classifiers in general.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.