Suppose I have the following output, alongside the ground-truth ($GT$), for two classifiers $C_1$ and $C_2$:

GT    C1    (C1 = GT)    C2     (C2 = GT) 
a     a         1         b         0 
b     a         0         b         1
b     b         1         b         1
a     a         1         a         1
a     b         0         a         1
b     b         1         a         0

Is the confusion table used for the McNemar's test the one built from columns $C_1$ and $C_2$, or from comparing columns $C_1 = GT$ and $C_2 = GT$?

My intuition says it is the later, but the material I've found about the subject was a little confusing.

  • $\begingroup$ McNemar's test is a test of the odds ratio in paired binary data. You shouldn't use that to compare classifiers. High OR != Good classification accuracy and vice versa. $\endgroup$ – AdamO Jun 21 '19 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.