Exact two sample proportions binomial test in R (and some strange p-values)

Question

I am trying to solve the following question:

Player A won 17 out of 25 games while player B won 8 out of 20 - is there a significant difference between both ratios?

The thing to do in R that comes to mind is the following:

> prop.test(c(17,8),c(25,20),correct=FALSE)

    2-sample test for equality of proportions without continuity correction

data:  c(17, 8) out of c(25, 20)
X-squared = 3.528, df = 1, p-value = 0.06034
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.002016956  0.562016956
sample estimates:
prop 1 prop 2 
  0.68   0.40 

So this test says that the difference is not significant at the 95% confidence level.

Because we know that prop.test() is only using an approximation I want to make things more exact by using an exact binomial test - and I do it both ways around:

> binom.test(x=17,n=25,p=8/20)

    Exact binomial test

data:  17 and 25
number of successes = 17, number of trials = 25, p-value = 0.006693
alternative hypothesis: true probability of success is not equal to 0.4
95 percent confidence interval:
 0.4649993 0.8505046
sample estimates:
probability of success 
                  0.68 

> binom.test(x=8,n=20,p=17/25)

    Exact binomial test

data:  8 and 20
number of successes = 8, number of trials = 20, p-value = 0.01377
alternative hypothesis: true probability of success is not equal to 0.68
95 percent confidence interval:
 0.1911901 0.6394574
sample estimates:
probability of success 
                   0.4 

Now this is strange, isn't it? The p-values are totally different each time! In both cases now the results are (highly) significant but the p-values seem to jump around rather haphazardly.

My questions

1. Why are the p-values that different each time?
2. How to perform an exact two sample proportions binomial test in R correctly?

Answer 1

If you are looking for an 'exact' test for two binomial proportions, I believe you are looking for Fisher's Exact Test. In R it is applied like so:

> fisher.test(matrix(c(17, 25-17, 8, 20-8), ncol=2))
    Fisher's Exact Test for Count Data
data:  matrix(c(17, 25 - 17, 8, 20 - 8), ncol = 2)
p-value = 0.07671
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
  0.7990888 13.0020065
sample estimates:
odds ratio 
  3.101466 

The fisher.test function accepts a matrix object of the 'successes' and 'failures' the two binomial proportions. As you can see, however, the two-sided hypothesis is still not significant, sorry to say. However, Fisher's Exact test is typically only applied when a cell count is low (typically this means 5 or less but some say 10), therefore your initial use of prop.test is more appropriate.

Regarding your binom.test calls, you are misunderstanding the call. When you run binom.test(x=17,n=25,p=8/20) you are testing whether proportion is significantly different from a population where the probability of success is 8/20. Likewise with binom.test(x=8,n=20,p=17/25) says the probability of success is 17/25 which is why these p-values differ. Therefore you are not comparing the two proportions at all.

Answer 2

There is a difference between two samples and a sample compared to a known hypothesis. So if someone flips a coin 100 times and gets heads 55 times and the hypothesis is a fair coin, versus two people flipping a coin of unknown fairness and one getting heads 55 times and the other 45 times. In the former case you are simply trying to identify if the flipper appears to be flipping a fair coin. In the latter, you are looking to see if they are flipping coins of the same fairness. You can see how if you are looking at each player against a known probability (45 vs. 50 and 55 vs. 50) is different than comparing them to each other (45 vs. 55).

Answer 3

The syntax of the binom.test is your successes within a number of trials compared to a population point estimate. Although you entered it as p=8/20, the calculation is as if that was a God-given absolute-truth 0.4 with zero variance around it. Or it is as if you were comparing player A's 17 wins out of 25 to player B's hypothetical 8 billion wins out of 20 billion games. However, prop.test compares the proportion of 17/25 with all its potential variance to the proportion of 8/20 with all of its own variance. In other words the variance around 0.7 (estimate of 17/25) and variance around 0.4 may bleed into one another with a resultant p=0.06.

Answer 4

First I would suggest that you want to do a continuity correction, since you are estimating a discrete distribution with a continuous (chi-square) distribution.

Second, it is important be clear on how the "experiment", if you will, was conducted. Were the number of games that each person played determined in advance (or in the vernacular of the industry, fixed by design)? If so, and further assuming each player's results are independent of the other, you are dealing with the product of 2 binomial distributions. If instead the number of games was free to vary (say for example, the number of games each person played were variables, based on the number of games each was able to complete in a fixed time frame), then you are dealing with a Multinomial or Poisson distribution.

In the second case the chi-square test (or what is the same thing, a z-test of difference in proportions) is appropriate, but in the former case it is not. In the first case, you really need to calculate the exact product of every possible binomial outcome for each player, and sum these probabilities for all occurrences that are equal to or less than the joint binomial probability of the outcomes that were observed (it is simply the product of the 2 binomials because each player's results are independent of the other player's results).

Recognize first that the central purpose of any hypothesis test is to calculate just how "rare" or unusual the specific outcome you have observed is, compared to all other possible outcomes. This is calculated by computing the probability of the outcome you have observed - given the assumption that the null hypothesis is true - summed together with all other possible outcomes of equal or lower probability.

Now it bares repeating that what we mean by "how rare" is " how low is the probability of observing the outcome obtained compared to all other possible outcomes?" Well, the probability of the specific outcome we have observed is 0.0679 * 0.0793 = 0.005115. Now consider a specific alternative outcome: it is certainly possible that player A could have won 7 of his 20 games and player B could have won 13 of his 25 games. The probability of this outcome is 0.004959. Note that this is LOWER than the probability of our observed outcome, so it should be included in the p-value. But look again: if you are deciding on which outcomes to include in your sum based on whether the difference in proportions exceeds to difference in proportions in our observed outcome, this probability will be excluded! Why? Because the difference in proportions for this specific outcome is less than the difference in proportions for our observed outcome. But this is not the proper focus - we must be concerned with the probability of this specific outcome and whether it is equal to or less than the probability of the outcome we have observed!

A good formal explanation of this can be found here:

http://data.princeton.edu/wws509/notes/c5.pdf

Please note specifically the statement on page 9 that "If the row margin is fixed and sampling scheme is binomial then we must use the product binomial model, because we can not estimate the joint distribution for the two variables without further information."