Coin toss experiments: When to use binomial, $\chi^2$ and Fisher's exact test?

Question

Let us say we have 2 independent trials with two different coins.

• In sample A, coin 1 is tossed 20 times and we record 6 heads and 14 tails.
• In sample B, coin 2 is flipped 30 times and we record 20 heads and 10 tails.

H0 is that both underlying distributions for experiments 1 and 2 are the same, or that both are "fair". Given the small sample size, what test would you deem appropriate here? Is a binomial test more exact here then the Fisher's / Chi2, and what is the reasoning

Many thanks for the help!

Answer 1

It seems to me that you are (or may be) asking several distinct questions.

Question 1: Test the null hypothesis that sample A and B are sampled from populations with the same ratio of heads/tails. Fishers exact test would answer this.

Question 2: Test the null hypothesis that the combined data from A and B are consistent with a fair coin (50% chance of heads, 50% chance of tails). You have 26 heads and 24 tails, very close to 50:50 so no test is needed. The binomial test could be used if you want to quantify the obvious answer.

Question 3: Test for each coin the hypothesis that the coin is fair so in the long run comes up heads half the time. The binomial test would answer this question, run twice, once for each coin.

Answer 2

Alternatively, you could think of this in a Bayesian context. Compute the posterior distribution of the probability of, say, a head separately for both experiments, perhaps using an uninformative prior distribution (I would suggest a $\textrm{Beta}(0,0)$ prior, which is improper but okay since your posterior will in this case be proper). Then simulate two vectors of the same length from each of those posteriors. Perform element-wise subtraction of one vector from the other to compute the posterior distribution of the difference in the probability of heads between the two coins. Then take the empirical quantiles of that difference to estimate a credible interval for the difference. As for computing the probability that the two coins are fair, you could compute the posterior probability that each coin's probability of heads is different from 0.5 by simply subtracting 0.5 from the vectors you computed above, then computing credible intervals by taking the desired empirical quantiles.