# p-value calculation for coin flipping

Here it mentioned we need to take into consideration of rare and rarer event (up to 12:28) for p-value calculation.

There is a problem: You observed 3 sequences of 5 flips of coin with at least one sequence was all heads.

The hypothesis to test is - whether the coin is bias. (The null hypothesis is the fair is not bias)

The probability calculation would be

 3 sequences of 5 flips of coin with at least one sequence was all heads
= prob of at least one sequence in 3 sequence was all heads
= 1- prob of no all heads in 3 sequence
= 1- (no all heads in 1 sequence)^3
= 1 - (1-all heads in 1 sequence)^3
= 1 - (1-(1/2)^5)^3 = 0.09


But what is the p-value?

• There is no p-value because you haven't articulated any testable hypotheses.
– whuber
Commented May 6, 2022 at 14:14
• @whuber thanks, and I have added the hypothesis Commented May 6, 2022 at 16:02
• That helps. But it occurs to me there can be no valid p-value in this setting, because you seem to be developing the test based on what you observed. Unless, before you flipped the coins, you had decided to base the test on the count of all-head sequences, this calculation will not be valid. It's an example of "HARKing" (Hypothesizing After Results are Known). Assuming the unlikely situation where you had proposed this statistic, you still need to frame an alternative hypothesis. See my post at stats.stackexchange.com/a/130772/919 for a detailed explanation.
– whuber
Commented May 6, 2022 at 16:06

You can change the null hypothesis, that you have a fair coin with $$p=0.5$$ for flipping heads, into an equivalent null hypothesis that you have a fair coin with $$p=0.5^5=0.0315$$ for flipping five heads in a series of five flips.

If you are making 3 of such series then the distribution under the null hypothesis is

A problem here is to define what observation is considered as a rare event. With the likelihood ratio test one would consider the observations with the lowest likelihood ratio, which are

$$LH = \begin{cases} \frac{0.90915}{1} & \approx & 0.90915 & \quad \text{if X=0}\\ \frac{0.08798}{0.4444} & \approx & 0.19769 & \quad \text{if X=1}\\ \frac{0.00284}{0.4444} & \approx & 0.00639& \quad \text{if X=2}\\ \frac{0.00003}{1} & \approx & 0.00003 & \quad \text{if X=3} \end{cases}$$

these denominators are the probabilities if the alternative hypothesis equals the maximum likelihood estimate.

The p-value is the probability of the observed likelihood ratio or higher and is the sum of the probabilities for $$X=1, X=2,X=3$$ and as you calculated $$\text{p-value} \approx 0.09085$$.

Note, the power of your test can ne very bad for specific values of the alternative hypothesis. One could say that your observation $$X=1$$ is rare given the null hypothesis, however, it will be much more rare when the null hypothesis is wrong and $$p<0.00335$$. So the power of your experiment is very bad and when $$p<0.00335$$ you will be unlikely to reject the null hypothesis.

A likelihood function gives a better view of the result:

Here you see that the fair coin at $$p=0.00325$$ is not very well supported. Although the same would be true for other values of $$p$$, the experiment is not very strong.

It isn't well-defined.

The p-value is the probability of getting a result "at least as extreme" as the observed one, conditional on the coin being fair.

But we don't have full information about what the observed result was - we are only told that at least one sequence was all heads.

Suppose that all 5 sequences had each consisted of 4 heads and 1 tail - would that be "more extreme"? The question vaguely implies that the answer is no, and that we only care about how many sequences were all heads - but that's absurd, since it implies that HHHHH HHHHH TTTTT is "more extreme" than HHHHT HHHHT HHHHT, even though the latter result is obviously stronger evidence that the coin is biased towards heads.