# sample size calculation for obtaining coin fairness

Suppose that someone gives you a coin with some unknown weighting (maybe it's a fair coin, or maybe it's just 25% likely to get heads, or etc). How many times would you have to flip the coin to determine, within some confidence, the weighting of the coin (the probability the coin will get heads in general)?

Edit: To clarify, I'm looking to see how the coin is loaded/weighted: for example, an "unloaded" coin would be weighted such that the probability of heads is 50%, whereas a "loaded" coin may have the heads probability at 70%, 90%, or etc. I want to know when I can stop flipping the coin with some level of confidence that the probability of heads is x% (where x% is calculated using the data, not assumed prior to having data).

Edit: To clarify further, I'll give an example: suppose I have some system that outputs a 0 or a 1 after each trial. After 5 trials, I end up with {01111}. So the two questions it raises are a) how do I find the probability that the next result is a 1, given only my 5 previous trials and b) how do I find the confidence of the calculation performed in a (based on the answers given so far, I'm guessing I can use a confidence as a stopping point, ie once a confidence level of 80% is reached I can stop doing more trials)?

• Do you suspect that the coin is biased? – Antoni Parellada Aug 15 '16 at 18:38
• My question before was meant to give me some time to find this video, which explains the Bayesian approach to the problem you are bringing up. – Antoni Parellada Aug 15 '16 at 18:56
• As opposed to the frequentist approach also explained here. – Antoni Parellada Aug 15 '16 at 18:56
• Is it a theoretical or practical problem? If practical, then the answer is: you do not need to throw it at all since biased coin is impossibility (cf stats.stackexchange.com/questions/153076/… ) – Tim Aug 15 '16 at 19:28
• @Tim, it's a theoretical problem: in reality, I'm working with something different from coins, but this was the simplest / most direct means by which I could phrase my problem. – mwarrior Aug 16 '16 at 19:19

It is fairly well explained here:

https://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair

Basically, using the Bayesian inference method and assuming an uniform prior distribution, which is reasonable (it represents maximum initial uncertainty about the fairness of the coin), the posterior probability for the actual probability $r$ of obtaining heads in a single toss after having observed $h$ number of heads and $t$ number of tails (therefore $n=h+t$ is the total number of tosses) is a Beta distribution with parameters $\alpha=h+1$ and $\beta=t+1$

$$f(r|H=h,T=t)=\frac{(h+t+1)!}{h!t!}r^h(1-r)^t$$

This will give you an idea of how r is distributed. The maximum-a-posteriori estimate (mode) is

$$r^*(h,t)=\frac{h}{h+t}$$

And the expected value is

$$E[r](h,t)=\frac{h+1}{h+t+2}$$

One can use the standard deviation as estimation of the uncertainty

$$\sigma(h,t)=\sqrt{\frac{(h+1)(t+1)}{(h+t+2)^2(h+t+3)}}$$

As you can see it doesn't depend just on the total number of tosses $h+t$ but also on $h$, so the criterion for a given confidence interval will be different depending on the sequence of results of the tosses.

• I guess that you meant that h/(h+t) is the maximum likelihood estimator? Expected value of beta-binomial model with "uniform" Beta(1,1) prior is given by you in next line: (h+1)/(h+t+2). – Tim Aug 15 '16 at 19:26
• What I meant is that h/(h+t) is the value of r that maximizes the posterior distribution. Since it is a skewed distribution the maximum and the mean take different values. It is usual to use the term "maximum a posteriori" in bayesian models, which literally means the value that produces the maximum of the posterior. – J. R. C. Aug 15 '16 at 23:00
• You are updating the Beta$(1,1)$ distribution. The posterior is a Beta$(h+1,t+1)$ distribution. It is therefore immediate that $\sigma(h, t)$ (which I write instead of "$\sigma(r)$" because it depends on $h$ and $t$) is the standard deviation of the Beta$(h+1, t+1)$ distribution, which has a simple closed formula. – whuber Aug 15 '16 at 23:33
• So you meant mode of beta distribution - I think that this needs clarifying the same as where the numbers come from (beta-binomial model not the binomial itself). Also: uniform prior is not "lack of prior"! – Tim Aug 16 '16 at 4:28
• Thanks for the comments, I improved the answer accordingly. – J. R. C. Aug 16 '16 at 13:52

I've been thinking about the Bayesian approach, but I believe now that it may end up being more theoretical or academic than intended in this very particular instance.

I wonder if what you are after (and I realize that I am working around some of the more specific questions at the end of your post) is to set up some statistical power calculations. You can do this with R, or online stats calculators, but the important concept is the flexibility in generating results that tell you in what percentage of cases you can expect to correctly call the coin biased depending on: 1. The number of tosses; 2. The degree of bias in the coin; and 3. The percentage of cases we are ready to accept as the risk of actually calling the coin biased when it is not (risk alpha).

Knowing that even with a fair coin you can get extremely surprising results, you can settle for a compromise that you will only call the coin "biased" if it goes beyond a threshold that guarantees you are going to make a mistake in only $\small 5\%$ of the times when the coin is in fact unbiased (risk alpha).

But you want to know what is in some respects a complementary idea: If the coin is biased, in what percentage of cases will you be able to make the call under the self-imposed constraints in the prior paragraph? In other words, the power. I have some notes in here if you are interested.

Evidently, your power to confidently reject the coin fairness will increase (will make itself more easily manifest) as you increase the number of tosses, or if you happen to be dealing with more extremely biased coins.

I am not going to re-invent the wheel by reformulating what is a very straightforward chunk of code that you can find in many guises, for example in this Berkeley University post, and that I'm pasting for ease of access:

set.seed(17) # Today's date.
coin.power = function(ntoss=100,nsim=1000,prob=.5){
lower = qbinom(.025,ntoss,.5)
upper = qbinom(.975,ntoss,.5)
rr = rbinom(nsim,ntoss,prob)
sum(rr < lower | rr > upper) / nsim
}
ntosses = c(10,100,200,500,600,800,1000,1500,2000,2500)
res = sapply(ntosses, coin.power, prob=.55)
names(res) = ntosses
res
10     100    200    500   600   800   1000  1500  2000  2500
0.032  0.133  0.259  0.634 0.653 0.799 0.867 0.969 0.994 0.999


In the case of a coin with a true biased probability of heads of $\small P(H)=55\%$, then, you can expect to reject correctly the hypothesis that the coin is fair $\small 99.4\%$ of the times if you toss the coin $\small 2,000$ times, and $\small 96.9\%$ of the times with $\small 1,500$ tosses.

If the coin is more biased, for example, $\small P(H)=70\%$, you can expect virtual certainty with $\small 200$ tosses:

res = sapply(ntosses,coin.power,prob=.70)
names(res) = ntosses
res
10    100   200   500   600   800   1000  1500  2000  2500
0.163 0.976 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000