# Power of Test in a Coin Toss Experiment

I am trying to understand the concept of "Power of Test" using the example of a coin experiment.

I have a coin that I want to prove is biased towards heads. I toss the coin 10 times, and get 9 heads and 1 tails. The probability of this happening to a fair coin is $$0.0107$$. On Excel, we can calculate this as: = BINOM.DIST(9, 10, 0.5, 0) + BINOM.DIST(10, 10, 0.5, 0). Or I can calculate this manually as $$\binom{10}{9}\cdot0.5^9\cdot0.5^1 + \binom{10}{10}\cdot0.5^{10}\cdot0.5^0 = 0.0107$$

This probability is so low, that I can say I have enough evidence to claim that the coin is not fair.

$$H_0$$ = Coin is Fair; $$H_A$$ = Coin is biased towards heads

Assuming $$H_0$$, the probability of at least 9 heads in 10 tosses - the $$p-value$$ - is so low that I reject $$H_0$$.

Is $$0.0107$$ the probability of incorrectly rejecting $$H_0$$? And $$1-0.0107$$ the probability of correctly not rejecting the null hypothesis (confidence)? How can I interpret this "confidence"?

And then, how do I calculate power of the test, and interpret it in this situation?

In your example, $$H_A$$ is a composite (rather than simple) hypothesis. In this case, we have to ask what exactly does it mean for $$H_A$$ to be true? The power here will be a function of the true value of $$p > 0.5$$.

For example, suppose that the true proportion of heads is $$p=0.7$$ and we use a testing procedure which rejects $$H_0$$ if there are at least $$9$$ out of $$10$$ heads. The size of this test is $$0.0107$$, as you computed in the original question. The power of this test is a function of $$p$$ and can be computed as $$P\left(\sum_{i=1}^{10}X_i \geq 9 \bigg| p=0.7\right) = \binom{9}{10} 0.7^9(1-0.7)^{10-1} + \binom{10}{10} 0.7^{10}(1-0.7)^{10-10} \approx 0.149.$$ So the power is just $$0.149$$, but this value will change for different values of $$p > 0.5$$.

True $$p$$ Power
0.51 0.0126
0.6 0.0464
0.7 0.149
0.9 0.736

For fixed $$p$$, you can increase the power by using a test based on a larger sample size. For example, suppose that now we will flip the coin $$n=40$$ times, and we will reject $$H_0$$ whenever there are at least $$26$$ heads. In R, the size of this test is given by

1 - pbinom(25, 40, 0.5)


which gives $$0.0403$$. The power (for given $$p>0.5$$) is found via

1 - pbinom(25, 40, p_A)

True $$p$$ Power
0.51 0.0526
0.6 0.317
0.7 0.807
0.9 0.999
• Thanks for the explaination! Using the null hypothesis rejection criteria of at least 9 heads of 10 toss, I will not be able to detect a small biasness towards heads (true p = 0.6, probability of correctly rejecting null hypothesis is low or, power = 0.0464), but if I do get at least 9 out of 10 heads, I can be fairly certain that the coin is biased (p-value = 0.0107). Instead, if I was trying to detect a large biasness (true p = 0.9), this criteria is much better (power = 0.736). Commented May 6 at 20:41
• If I change the criteria to at least 6 heads of 10 toss, I will be able to detect the true p = 0.6 with power = 0.63, but I can't reject null hypothesis as p-value = 0.38. I can extend this logic and design a test that can do both: a rejection criteria of at least 536 heads of 1000 toss allows me to detect true p = 0.6 with a power of 0.99, while still giving strong evidence of biasness (p-value = 0.0105). Commented May 6 at 20:51
• @DragonClaw You have to be careful with terminology. What you are calling p-value is actually called the "size" of the test (also called $\alpha$, the significance level, or the type I error rate). The P-value is calculated from the data, and we reject if $p < \alpha$. Commented May 6 at 22:13
• @DragonClaw. But yes -- you are generally thinking about this in a useful way. There is always a tradeoff between the size and power of a test (or equivalently, a tradeoff between the Type I and Type II errors). The best you can do to ensure satisfactory values for both, is to increase the sample size. Commented May 6 at 22:15