Coin toss experiments: When to use binomial, $\chi^2$ and Fisher's exact test?

Let us say we have 2 independent trials with two different coins.

• In sample A, coin 1 is tossed 20 times and we record 6 heads and 14 tails.
• In sample B, coin 2 is flipped 30 times and we record 20 heads and 10 tails.

H0 is that both underlying distributions for experiments 1 and 2 are the same, or that both are "fair". Given the small sample size, what test would you deem appropriate here? Is a binomial test more exact here then the Fisher's / Chi2, and what is the reasoning

Many thanks for the help!

• Have you tried implementing the two tests? Look at their p-values and think about what you see. – user25658 Sep 4 '13 at 14:50
• @Bab Although you probably didn't intend this, it would be possible--perhaps likely--for many readers to interpret your comment as a recommendation that we should select statistical tests according to whichever one produces a p-value that is most satisfactory to us. As you know that is invalid (and proceeding this way without reporting which tests one examined is considered to be unethical by many). – whuber Sep 4 '13 at 18:47
• Just want to make sure you're clear that "both are the same" and "both are fair" are very different questions. – Glen_b -Reinstate Monica Sep 4 '13 at 23:48

It seems to me that you are (or may be) asking several distinct questions.

Question 1: Test the null hypothesis that sample A and B are sampled from populations with the same ratio of heads/tails. Fishers exact test would answer this.

Question 2: Test the null hypothesis that the combined data from A and B are consistent with a fair coin (50% chance of heads, 50% chance of tails). You have 26 heads and 24 tails, very close to 50:50 so no test is needed. The binomial test could be used if you want to quantify the obvious answer.

Question 3: Test for each coin the hypothesis that the coin is fair so in the long run comes up heads half the time. The binomial test would answer this question, run twice, once for each coin.

• There is also the possibility of "Test the null hypothesis that sample A and B are each sampled from populations with the same ratio of heads/tails and that ratio is $\frac12$" – Henry Sep 4 '13 at 17:57
• How did you come up with this interpretation of Question 2? I cannot find any place where the original text asserts the null is that the combined results look like they came from a fair coin. – whuber Sep 4 '13 at 18:44
• @whuber. The OP said he wanted to test "that both are fair", so I took that to be combined. But it probably makes more sense to ask if each is fair, so that would require two binomial tests. I'll reword. – Harvey Motulsky Sep 4 '13 at 23:14

Alternatively, you could think of this in a Bayesian context. Compute the posterior distribution of the probability of, say, a head separately for both experiments, perhaps using an uninformative prior distribution (I would suggest a $\textrm{Beta}(0,0)$ prior, which is improper but okay since your posterior will in this case be proper). Then simulate two vectors of the same length from each of those posteriors. Perform element-wise subtraction of one vector from the other to compute the posterior distribution of the difference in the probability of heads between the two coins. Then take the empirical quantiles of that difference to estimate a credible interval for the difference. As for computing the probability that the two coins are fair, you could compute the posterior probability that each coin's probability of heads is different from 0.5 by simply subtracting 0.5 from the vectors you computed above, then computing credible intervals by taking the desired empirical quantiles.