Let $C_1$, $C_2$ be two binary classifiers, which are used to classify some data (images, videos, etc) to $30$ different classes, using an one-against-all approach. Then, we have two $30$-dimensional vectors containing the performances of the above classifiers in terms of average precision (AP). That is, $$ \mathbf{v}_1 = \big(p_{1,1},\ldots,p_{1,30} \big)^\top, $$ and $$ \mathbf{v}_2 = \big(p_{2,1},\ldots,p_{2,30} \big)^\top, $$ where $p_{i,j}\in[0,1]$, for all possible $i$, $j$. Let $\bar{v}_1$, $\bar{v}_2$ be the mean values of $\mathbf{v}_1$ and $\mathbf{v}_2$, respectively.

Let's assume that $\bar{v}_2>\bar{v}_1$. I would like to test if this is statistically significant.

What I have tried so far is the sign test, but I have been advised to also check the McNemar's test. However, I cannot see how I could possibly use this test.

What would you suggest me to do? Is there any particular reason why I should prefer McNemar's test over the sign test? How could I use the McMenar's test?

Thanks in advance!

  • $\begingroup$ Why not the paired t-test? That's what I'd use for comparing performance over N problems. $\endgroup$
    – PawelP
    Commented Jun 11, 2015 at 15:07

2 Answers 2


McNemar's Test

I don't think you can apply McNemar's test on the vector of precisions. Instead, you would apply it to the 2x2 contigency table of cross-classification of test items:

$$ \begin{array}{lll} & \mbox{Correct in C1} & \mbox{Incorrect in C1} \\ \mbox{Correct in C2} & m_{11} & m_{12} \\ \mbox{Incorrect in C2} & m_{21} & m_{22} \\ \end{array} $$ formed over all 30 classes (which would give you the average over classes, weighted by their incidence in the test data). Edit: This table is formed using the performance each classifier has on each particular item, and assumes that the same items are shown to each classifier. So the vectors of precisions $\mathbf v_1$,$\mathbf v_2$ will not suffice. McNemar's test is then testing that the probability of C1 being incorrect while C2 is correct ($p_{12}$) equals the probability of C1 being correct while C2 is incorrect ($p_{21}$). See wikipedia.

Comment on McNemar's Test

McNemar's test will undoubtedly be more sensitive than using the precision vectors, because it is controlling for the variability due to the underlying difficulty of each item being classified. If you reject then you know that the misclassification rates differ by classifier, averaging over the background prevalence of the 30 categories. This is testing something a little bit different than the precisions, but seems pretty reasonable as long as each classifier must return a label from one of the 30 categories (ie, it can't refuse to classify an item).

Alternatives not requiring item-level cross tabs

If you don't have the 2x2 cross-classification table, then you would indeed be forced to do something with your vector of precisions. A couple issues here:

  1. Your vectors $\mathbf{v}_1$ $\mathbf{v}_2$ probably have different variances, because each component $v_{ij}$ is an average over a varying number of items, so will have variances proportional to $n_{ij}^{-1/2}$, where $n_{ij}$ is the number of detected positives for class $i$, classifier $j$. So unless you account for this, you will not have correct inference (a test will not have the correct rejection rate).
  2. As binomial proportions, the variances of $\mathbf{v}_1$ $\mathbf{v}_2$ also depend on the usual fashion on $v_{ij}(1-v_{ij})$.

1 and 2 suggest that if you indeed cannot recover the pairing between the classifiers, that you should use binomial regression on the model: $$E ( Y_{ij} | n_{ij} ) = \alpha + \beta x_j$$ where $Y_{ij}, n_{ij}$ is the number of true positives and total positives, respectively, for class $i$, classifier $j$, $\alpha$ is an intercept (giving the precision for classifier 1), $x_j=0$ for classifier 1 and $x_j=1$ for classifier 2, and $\beta$ gives the difference in precision between the two classifiers. In R, this would look like this:

glm(cbind(Ncorrect, Nincorrect) ~ classifier,
    data=TableOfClassifierPerformanceByClass, family='binomial')

Comment binomial regression

This would be a good approach if you do not have access to the item-level crosstabs (joint confusion matrix), or if you need to test the precision per se.

Tests on $\mathbf v_1$ $\mathbf v_2$ alone

You could also use a non-parametric test on the vector of precisions alone (thus ignoring the information you have about how it was generated). These may be considerably less powerful than the options identified above (since you effectively only have a sample size of 30, and you don't get to leverage the knowledge that a precision formed over many hundreds of items has small sampling variability).

  • $\begingroup$ Many thanks for your answer! I am not sure I understand exactly what you suggest, but I will try to. What's your opinion on the usage of the sign test over the vectors of the average precision's? Thanks again! $\endgroup$ Commented Dec 17, 2014 at 8:53
  • $\begingroup$ It's a less powerful method than the options that I discuss, since it ignores the number of observations that were used to establish $\mathbf{v}_1$ and $\mathbf{v}_2$. If the precision difference between the classifiers is large, then it might not matter. I don't think it would lead to you conclude that there is a difference between classifiers when there is no difference. $\endgroup$
    – Andrew M
    Commented Dec 17, 2014 at 9:22
  • $\begingroup$ I see, that's what I was thinking about the sign test. So, concerning the McNemar's test, do you suggest I would take it pairwise; for each class? And then, how could I conclude that the difference is significant or not, over the 30 classes? $\endgroup$ Commented Dec 17, 2014 at 9:32
  • $\begingroup$ Also, the sign test gives inference on the median of $\mathbf{v}_2 - \mathbf{v}_1$, not the mean, as you suggest in your initial question. If you reject, it's saying that the median precision (across the 30 categories), in the average test item (that is called a positive for a category and classifier), is larger in classifier 2. $\endgroup$
    – Andrew M
    Commented Dec 17, 2014 at 9:32
  • $\begingroup$ So, this is not something desirable? I mean, talking about the median of $\mathbf{v}_2-\mathbf{v}_1$? $\endgroup$ Commented Dec 17, 2014 at 9:35

I would recommend using the paired T-test: http://www.graphpad.com/quickcalcs/ttest1.cfm (an online tool - select paired t-test) http://www.statstutor.ac.uk/resources/uploaded/paired-t-test.pdf (a few words about the test)

It's a natural way of comparing corresponding results. The paired t-test in this case would operate on V1-V2 differences, seeing whether there is some consistency in one method outperforming the other.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.