1
$\begingroup$

I am trying to understand the concept of "Power of Test" using the example of a coin experiment.

I have a coin that I want to prove is biased towards heads. I toss the coin 10 times, and get 9 heads and 1 tails. The probability of this happening to a fair coin is $0.0107$. On Excel, we can calculate this as: = BINOM.DIST(9, 10, 0.5, 0) + BINOM.DIST(10, 10, 0.5, 0). Or I can calculate this manually as $\binom{10}{9}\cdot0.5^9\cdot0.5^1 + \binom{10}{10}\cdot0.5^{10}\cdot0.5^0 = 0.0107$

This probability is so low, that I can say I have enough evidence to claim that the coin is not fair.

$H_0$ = Coin is Fair; $H_A$ = Coin is biased towards heads

Assuming $H_0$, the probability of at least 9 heads in 10 tosses - the $p-value$ - is so low that I reject $H_0$.

Is $0.0107$ the probability of incorrectly rejecting $H_0$? And $1-0.0107$ the probability of correctly not rejecting the null hypothesis (confidence)? How can I interpret this "confidence"?

And then, how do I calculate power of the test, and interpret it in this situation?

Thanks for your help!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In your example, $H_A$ is a composite (rather than simple) hypothesis. In this case, we have to ask what exactly does it mean for $H_A$ to be true? The power here will be a function of the true value of $p > 0.5$.

For example, suppose that the true proportion of heads is $p=0.7$ and we use a testing procedure which rejects $H_0$ if there are at least $9$ out of $10$ heads. The size of this test is $0.0107$, as you computed in the original question. The power of this test is a function of $p$ and can be computed as $$P\left(\sum_{i=1}^{10}X_i \geq 9 \bigg| p=0.7\right) = \binom{9}{10} 0.7^9(1-0.7)^{10-1} + \binom{10}{10} 0.7^{10}(1-0.7)^{10-10} \approx 0.149.$$ So the power is just $0.149$, but this value will change for different values of $p > 0.5$.

True $p$ Power
0.51 0.0126
0.6 0.0464
0.7 0.149
0.9 0.736

For fixed $p$, you can increase the power by using a test based on a larger sample size. For example, suppose that now we will flip the coin $n=40$ times, and we will reject $H_0$ whenever there are at least $26$ heads. In R, the size of this test is given by

1 - pbinom(25, 40, 0.5)

which gives $0.0403$. The power (for given $p>0.5$) is found via

1 - pbinom(25, 40, p_A)
True $p$ Power
0.51 0.0526
0.6 0.317
0.7 0.807
0.9 0.999
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the explaination! Using the null hypothesis rejection criteria of at least 9 heads of 10 toss, I will not be able to detect a small biasness towards heads (true p = 0.6, probability of correctly rejecting null hypothesis is low or, power = 0.0464), but if I do get at least 9 out of 10 heads, I can be fairly certain that the coin is biased (p-value = 0.0107). Instead, if I was trying to detect a large biasness (true p = 0.9), this criteria is much better (power = 0.736). $\endgroup$
    – DragonClaw
    Commented May 6 at 20:41
  • $\begingroup$ If I change the criteria to at least 6 heads of 10 toss, I will be able to detect the true p = 0.6 with power = 0.63, but I can't reject null hypothesis as p-value = 0.38. I can extend this logic and design a test that can do both: a rejection criteria of at least 536 heads of 1000 toss allows me to detect true p = 0.6 with a power of 0.99, while still giving strong evidence of biasness (p-value = 0.0105). $\endgroup$
    – DragonClaw
    Commented May 6 at 20:51
  • $\begingroup$ @DragonClaw You have to be careful with terminology. What you are calling p-value is actually called the "size" of the test (also called $\alpha$, the significance level, or the type I error rate). The P-value is calculated from the data, and we reject if $p < \alpha$. $\endgroup$
    – knrumsey
    Commented May 6 at 22:13
  • $\begingroup$ @DragonClaw. But yes -- you are generally thinking about this in a useful way. There is always a tradeoff between the size and power of a test (or equivalently, a tradeoff between the Type I and Type II errors). The best you can do to ensure satisfactory values for both, is to increase the sample size. $\endgroup$
    – knrumsey
    Commented May 6 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.