I flip a coin 20 times and get 14 heads. I want to calculate the p-value of the hypothesis that my coin is fair.

What probability should I calculate?

Wikipedia says that I need to calculate the probability to get 14 or more heads in 20 flips.

Why is it 14 "or more"? Why not 14 or less?

  • 3
    $\begingroup$ Haven't read the Wikipedia entry (which one?), but: this problem is not fully specified. In the absence of more information I would say you should calculate P(H>=14 OR H<=6), i.e. a two-tailed test (the probability, under the null hypothesis (H=10) that the value deviates by as much or more than observed in either direction from the expected value under the null). $\endgroup$ – Ben Bolker Dec 16 '10 at 16:31
  • $\begingroup$ I suggest you have a look at "L. D. Brown, T. T. Cai and A. DasGupta, Interval Estimation for a Binomial Proportion, Statistical Science, Vol. 16, pp. 101-117, 2001". It includes a very nice discussion of interval estimation for the Binomial distribution. $\endgroup$ – emakalic Dec 16 '10 at 23:20

It's neither because the alternative to being fair is that the coin favors heads or tails.

You are free to invent any test you like. For example, I could (idiosyncratically) decide the coin is unfair if and only if the number of heads is either 6 or 15 (the "critical region"), because this event occurs with only 5% chance when the coin is fair. The key question is how well does a test perform. The Neyman-Pearson lemma shows that this particular one I just invented is a poor test. A good test is one whose critical region not only is unlikely when the null is true, but is also highly likely when the null is false.

There is no one best critical region for this kind of two-sided test, but a reasonable compromise is to adopt a procedure that will detect deviations from fairness in both directions. That suggests a critical region that contains the most extreme possibilities: a bunch near 0 and a bunch near 20. A good choice at the 5% level is to consider any outcome of 15 or more or 5 or less to be significant.

Let us then adopt the best symmetric test for a fair coin. This means we want the critical region to include $20-i$ heads whenever it includes $i$ heads (that is, $20-i$ tails). This treats heads and tails on an equal footing. It is only in the context of a particular test (like this one) that a p-value has any meaning. The p-value corresponding to an outcome of 14 is, by definition, the smallest significance of any such test that includes 14 in its critical region. By symmetry it must include 20 - 14 = 6 and by the Neyman-Pearson lemma it must include all values larger than 14 and all values less than 6. The chance of this under the null is 11.53%. This chance increases uniformly as the chance of heads deviates more and more from 1/2 in either direction.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.