How to test if all coins have the same bias

I have data from a coin flipping experiment. I flipped $N$ coins $M$ times and want to know if the probability of heads is the same across all the coins or if it varies. It might be that one coin has a different bias than all the others or that there $k$ sets of coins each with a unique bias. I am having trouble framing the problem in a way that I can see what statistical test to use.

• Hint: Presumably you hope each coin will have approximately (in some sense) the same proportion of heads and tails when the null hypothesis is true. Perhaps a contingency table showing numbers of heads and tails by coin might help you estimate how many you would have expected of each and help you calculate how big the difference of your observations is from this expectation Commented Aug 9, 2017 at 18:18

Perhaps an good first approach is to produce a confidence interval for the probability of heads for each coin and see if there is overlap.

As a working example, I simulated some data for $N=10$ coins, each flipped $M=500$ times. In the simulation, each coin is "fair" with $p_i = 0.5$ except for the third coin which has $p_3 = 0.7$.

N <- 10
M <- 500
p_i <- rep(0.5, N)
p_i[3] <- 0.7
Y <- rbinom(N, M, p_i)


Using R, we can readily compute the confidence intervals for $p_i$ with confidence level $\gamma = 1 - 0.05 = 0.95$. The interval for the first coin can be extracted as follows.

 conf <- binom.test(Y[1], M, conf.level=0.95)$conf.int  We can repeat this for each coin, an produce a plot of the confidence intervals. The sample size appears to be large enough, for us to pick up on the fact that the third coin has a higher probability of heads than the others. There is a problem with this procedure. We are making multiple comparisons and not considering this in the analysis. The idea is, if you flip enough different coins, one of them is bound to be "different" from the others eventually, based solely on chance. We can fix this (possibly at the cost of less power) with the Bonferonni Correction. Without going into details, we need to adjust the confidence level according the number of different coins flipped. $$\gamma = 1 - \frac{\alpha}{N} = 0.995$$ We can repeat the procedure, this time putting, conf <- binom.test(y[1], M, conf.level=0.995)$conf.int


Notice that the confidence intervals are wider this time. These intervals define a Joint Confidence Region, which is to say that these intervals hold simultaneously with (at least) 95% confidence.

Even with the larger intervals, we are able to correctly identify that the third coin has a different probability of heads than the others.

In summary, you will be able to pick up on differences more effectively with a larger sample size $M$. This graphical approach will allow you to visualize single coins or groups of coins with different "bias" than others.

An example, with the Bonferonni Correction demonstrating the case questioned in the comment. Here, $\epsilon=0.1$ so that the first 5 coins have $p_i = 0.4$ and the next 5 coins have $p_i = 0.6$.

• I am not sure how/if this captures a bimodal distribution where $N/2$ coins have a probability of heads of $p+\epsilon$ and the rest have a probability of heads of $p-\epsilon$. Commented Aug 9, 2017 at 18:52
• Sure it does. See update. If $\epsilon$ is very small, then no statistical procedure will capture the difference, unless the sample size is enormous. Commented Aug 9, 2017 at 18:53
• If I am following the answer, if the difference was smaller so the CIs all overlapped you would conclude that all the coins are the same, but it seems like a biomodal distribution would be unlikely if they were all the same. Maybe what I want to test is if the data is binomial. Commented Aug 9, 2017 at 19:00
• Formally, you would "lack evidence to conclude the coins were different". Detecting small differences in probability is going to be difficult regardless of method. The Binomial assumption should be quite valid, for each coin. If you are interested in the difference in probabilities, you should use a method like this one, or a contingency table as recommended by @Henry in his comment. Commented Aug 9, 2017 at 19:06