Suppose my sample comprises 350 instances. Each instance has two independent binary (pos and neg) evaluations, A and B. If my alternative hypothesis is that A is more often positive than B, how would I express whether the difference is statistically significant?

To be more precise on the data: This question came to mind when looking at patient blood serum data. These are 350 blood samples positive against X. Each sample may or may not also be positive against A or B, whereas A and B are both more often positive if X is positive, but biologically are not dependent on one another.

So my initial thought was to look whether A is more often positive than B. In the example it was, A is positive in ~80% and B in ~20%. So I thought about assessing this difference for significance and this is where I got stuck. :)

  • 1
    $\begingroup$ X seems irrelevant to your question of whether A is more often positive than B. Unless perhaps you might wish to stratify by X, i.e. ask (1) whether A is more often positive than B when X is positive, then (2) whether A is more often positive than B when X is negative ?? $\endgroup$
    – onestop
    Nov 25 '10 at 19:52
  • $\begingroup$ X seemed irrelevant to my question, but maybe it's not. Still, the question is more along the line of tossing two coins. If one turns up heads 80% of the time and the other 20% of the time, how can I asses this difference. With the coin I have an assumed probability of 0.5, so maybe this is solved differently, I don't have an assumed probability for my example, unless I deduce one from all samples, whether or not X is positive. $\endgroup$
    – Pascal
    Nov 25 '10 at 20:24

You are conducting a one-sided test of a difference of proportions. Because all four outcomes--A, not A, B, not B--occur often (70 times or more in this case), the Normal approximation to the Binomial distribution will be just fine. Let $a$ be the number of occurrences of A, $b$ the number of occurrences of B, and $n$ the total sample (about 350). Under the null hypothesis $a = b$ the variance of a single observation is estimated with the combined data, $s^2 = (a+b)/(2n) \cdot (1 - (a+b)/(2n))$. The variances of A and B are estimated as $s^2/n$. The test statistic therefore is

$$z = \frac{a/n - b/n}{\sqrt{s^2/n + s^2/n}}.$$

The p-value equals $1 - \Phi^{-1}(z)$ where $\Phi^{-1}$ is the percentage point function for the standard normal distribution.

For example, with $n = 350$, $a = 280$, and $b = 70$, we estimate $a$ as 0.8, $b$ as 0.2, and $s^2$ as 0.25, giving $z = 15.87$. Obviously that's not due to chance. The result will be equally strong and obvious for any values of $a$, $b$, and $n$ anywhere close to these.

  • $\begingroup$ In the denominator, isn't it $s^2/n$ instead? Also, do you need to assume independence of A and B for this test? $\endgroup$
    – caracal
    Nov 28 '10 at 14:35
  • $\begingroup$ @caracal Yes, thanks for catching that error. I have fixed it. (The calculations in the example were nevertheless correct.) Independence of A and B is asserted by the OP in the second sentence of the question. $\endgroup$
    – whuber
    Nov 28 '10 at 19:19

Your addendum suggests that A and B are dependent samples since they come from the same "instance", i.e., patient. In that case, I propose the McNemar-Test which tests for a (two-sided) hypothesis of unequal proportions:

> N    <- 350
> A    <- factor(rbinom(N, size=1, p=0.6), labels=c("pos", "neg"))
> B    <- factor(rbinom(N, size=1, p=0.4), labels=c("pos", "neg"))
> cTab <- table(A, B)
> mcnemar.test(cTab, correct=FALSE)   # no continuity correction
        McNemar's Chi-squared test
data:  cTab 
McNemar's chi-squared = 29.1236, df = 1, p-value = 6.79e-08

# alternative - Monte-Carlo permutation test
> library(coin)          # for symmetry_test()
> symmetry_test(cTab, teststat="quad", distribution=approximate(B=9999))
        Approximative General Independence Test
data:  response by groups (A, B) 
         stratified by block 
chi-squared = 29.1236, p-value < 2.2e-16
  • $\begingroup$ Thanks! That's very good to get me started, I'll see how it goes and then mark it as correct answer when I'm capable to judge this. :) Note I also edited the question a little. $\endgroup$
    – Pascal
    Nov 25 '10 at 17:46
  • 1
    $\begingroup$ I think it's now clear that McNemar's test answers your original question. Fisher's exact test is not relevant here. $\endgroup$
    – onestop
    Nov 25 '10 at 19:53
  • $\begingroup$ I can use both tests to test the relationship between A and B, Fisher's test tells me whether these are independent indeed, McNemar's seems to be more appropriate when comparing the same measure before and after an event. What I intended to test is more whether A or B have a higher probability, e.g. if I toss two coins 350 times and one turns up heads 80% of the time and the other 20%, how can I calculate the significance of this difference. Sorry for my unclear question. $\endgroup$
    – Pascal
    Nov 25 '10 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.