McNemar's test is a special example of the binomial sign test, but the ('vanilla') sign test suffers from bias due to ignoring differences equal to zero.

McNemar's test statistic is given by:

$\chi^{2} = \frac{\left(|r-s|-1\right)^{2}}{r+s}$, where $r$ and $s$ are the counts of discordant pairs (0,1) versus (1,0), distributed $\chi^{2}$ with 1 degree of freedom under the null hypothesis.

I am having a hard time parsing Sribney on the sign test:

The test statistic for the sign test is the number $n_{+}$ of observations greater than zero. Assuming that the probability of an observation being equal to zero is exactly zero, then, under the null hypothesis, $n_{+} \sim \text{Binomial}(n, p=\frac{1}{2})$, where $n$ is the total number of observations. But what do we do if we have some observations that are zero?

Fisher’s Principle of Randomization

We have a ready answer to this question if we view the test from the perspective of Fisher’s Principle of Randomization (Fisher 1935). Fisher’s idea (stated in a modern way) was to look at a family of transformations of the observed data such that the a priori likelihood (under the null hypothesis) of the transformed data is the same as the likelihood of the observed data. The distribution of the test statistic is then produced by calculating its value for each of the transformed “randomization” data sets, considering each data set equally likely.

For the sign test, the “data” are simply the set of signs of the observations. Under the null hypothesis of the sign test, $P(X_{i}>0)= P(X_{i}<0)$, so we can transform the observed signs by flipping any number of them and the set of signs will have the same likelihood. The $2^{n}$ possible sign changes form the family of randomization data sets. If we have no zeros, this procedure again leads to $n_{+} \sim \text{Binomial}(n, p=\frac{1}{2})$.

If we do have zeros, changing their signs leaves them as zeros. So if we observe $n_{0}$ zeros, each of the $2^{n}$ sign-change data sets will also have $n_{0}$ zeros. Hence, the values of $n_{+}$ calculated over the sign-change data sets range from 0 to $n-n_{0}$, and the “randomization” distribution of $n_{+}$ is $n_{+} \sim \text{Binomial}(n-n_{0}, p=\frac{1}{2})$.

Because this seems to be saying go ahead and ignore zeros. But then later in the paper, Sribney provides an adjustment for the sign-rank test that accounts for zeros just along the lines I am wondering about:

The adjustment for zeros is the change in the variance when the ranks for the zeros are signed to make $r_{j}=0$; i.e., the variance is reduced by $\frac{1}{4}\sum_{i=1}^{n-{0}}{i^{2}}=n_{0}\frac{\left(n_{0}+1\right)\left(2n_{0}+1\right)}{24}$.

Should I instead be asking whether or not to apply the signed-rank test to individually-matched case control data?

A simple made up example will illustrate why ignoring zeros presents a problem. Imagine you've paired data with no differences equal to zero (this would correspond to data for a McNemar's test with only discordant pairs present). With a sample size of, say, 20, you find 15 positive signs of differences and 5 negative signs of differences, and conclude significant difference. Now imagine that you have 1000 observed differences equal to zero in addition to those 15 positive and 5 negative signs of differences: now you conclude difference is not significant. If McNemar's test is conducted on 1020 pairs, 1000 of which are zeros, and with discordant pairs of 15 and 5, we should not reject the null hypothesis (e.g. at $\alpha = 0.05$).

There is an adjustment to the sign test to correct for observed zero differences based upon Fisher’s "Principle of Randomization" (Sribney, 1995).

Is there a way of improving on McNemar's test that addresses the effect of observed zero differences (i.e. by accounting for number of concordant pairs relative to number of discordant pairs)? How? What about for the asymptotic z approximation for the sign test?


Sribney WM. (1995) Correcting for ties and zeros in sign and rank tests. Stata Technical Bulletin. 26:2–4.

  • 1
    $\begingroup$ I've used to think that McNemar = Sign in case of dichotomous data, - at least, SPSS seems to always return the same result for them. If that is so, then everything invented to enhance one of the two tests may be confidently applied to the other. $\endgroup$ – ttnphns Jun 12 '14 at 17:13
  • $\begingroup$ @ttnphns Yes... that is my thought also. Now I am trying to figure out how. :) $\endgroup$ – Alexis Jun 12 '14 at 17:15
  • $\begingroup$ BTW, it could be really great if you present the authors arguments why a correction for ties is needed in sign/rank tests. I've just googled and couldn't find the paper in the web so far. $\endgroup$ – ttnphns Jun 12 '14 at 17:21
  • $\begingroup$ I've just looked in "SPSS algorithms" and I confirm that these are identical tests, both in exact and in asymptotic versions. They actually have the same formulas. $\endgroup$ – ttnphns Jun 12 '14 at 18:03
  • $\begingroup$ @ttnphns Good idea. And DONE! $\endgroup$ – Alexis Jun 12 '14 at 19:24

E.L. Lehmann, J.P. Romano. Testing Statistical Hypotheses. 3rd ed. Springer, 2005. P. 136:

P(-), P(+) and P(0) denote the probabilities of preference for product [A over B, B over A, or A=B, tie], ... The hypothesis to be tested H0: P(+)=P(-) ... The problem reduces to that of testing the hypothesis P=1/2 in a binomial distribution with n-z [n = sample size, z = number of ties] trials ... The unbiased test is obtained therefore by disregarding the number of cases with no preference (ties), and applying the sign test to the remaining data.

The power of the test depends strongly on P(0) ... For large P(0), the number n-z of trials in the conditional [on z] binomial distribution can be expected to be small, and the test will thus have little power ... A sufficiently high value of P(0), regardless of the value of P(+)/P(-), implies that the population as a whole is largely indifferent with respect to the product.

As an alternative treatment of ties, it is sometimes proposed to assign each tie at random (with probability 1/2 each) to either plus or minus ... The hypothesis H0 becomes P(+)+1/2P(0)=1/2 ... This test can be viewed also as a randomized test ... and it is unbiased for testing H0 in its original form ... [But] Since the test involves randomization other than on the boundaries of the rejection region, it is less powerful than the [original test disregarding ties]..., so that the random breaking of ties results in a loss of power.

| cite | improve this answer | |

I don't see how this would be helpful, or even possible. McNemar's test only uses the discordant pairs. The Wikipedia page states:

The McNemar test statistic is: $$ \chi^2 = {(b-c)^2 \over b+c}. $$

I have a lengthy explanation of McNemar's test here: What is the difference between McNemar's test and the chi-squared test, and how do you know when to use each? The whole post may be of value, but borrowing the example there:

\begin{array}{rrrrrr} & &{\rm After} & & & \\ & &{\rm No} &{\rm Yes} & &{\rm total} \\ {\rm Before}&{\rm No} &1157 &35 & &1192 \\ &{\rm Yes} &220 &13 & &233 \\ & & & & & \\ &{\rm total} &1377 &48 & &1425 \\ \end{array}

McNemar's test is the binomial test of $220/(220+35)$; the concordant pairs (i.e., $1157$, and $13$) don't show up.

| cite | improve this answer | |
  • $\begingroup$ In SPSS, both tests share the same formulas and are computed with continuety correction, (|b-c|-1)^2 the numerator. $\endgroup$ – ttnphns Jun 12 '14 at 18:07
  • 1
    $\begingroup$ This is just the basic (conceptual) form. Using the continuity correction is fine. It is also the default in R. If the counts are small, you can use the exact binomial test instead of the chi-squared approximation (w/ or w/o the continuity correction). At any rate, you are only using the discordant pairs. $\endgroup$ – gung - Reinstate Monica Jun 12 '14 at 18:11
  • $\begingroup$ But to say honestly, I don't know why the concordant counts (and in general, ties) are not used in most nonparametric tests. Intuitively it looks unnatural. What math, probability or philisophical reasons might be behind it? $\endgroup$ – ttnphns Jun 12 '14 at 18:18
  • $\begingroup$ The reasoning is in my linked answer. Briefly, it's because they are uninformative & would be being counted twice. $\endgroup$ – gung - Reinstate Monica Jun 12 '14 at 18:53
  • $\begingroup$ @gung I am quite aware that McNemar's test only uses concordant pairs (just like the 'vanilla' sign test only uses positive and negative signs). The vanilla McNemar's test is the sign test (in disguise). There is an adjustment to the sign test to correct the bias resulting from ignoring zeros. My question was how to incorporate that in McNemar test format. $\endgroup$ – Alexis Jun 12 '14 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.