You observe k heads out of n tosses. Is the coin fair?

Question

I was asked this question with $(n, k) = (400, 220)$ in an interview. Is there a "correct" answer?

Assume the tosses are i.i.d. and the probability of heads is $p=0.5$. The distribution of the number of heads in 400 tosses should then be close to Normal(200, 10^2), so that 220 heads is 2 standard deviations away from the mean. The probability of observing such an outcome (i.e. more 2 SDs away from the mean in either direction) is slightly less than 5%.

The interviewer told me, essentially, "if I observe something >= 2 SDs from the mean, I conclude that something else is going on. I would bet against the coin being fair." That's reasonable -- after all, that's what most hypothesis tests do. But is that the end of the story? For the interviewer that seemed to be the "correct" answer. What I'm asking here is whether some nuance is justified.

I couldn't help but point out that deciding that the coin is not fair is a bizarre conclusion in this coin-tossing context. Am I right to say that? I'll try and explain below.

First of all, I -- and I would assume most people as well -- have a strong prior about coins: they're very likely to be fair. Of course that depends on what we mean by fair -- one possibility would be to define "fair" as "having a probability of heads 'close' to 0.5, say between 0.49 and 0.51."

(You could also define 'fair' as meaning that the probability of heads is exactly 0.50, in which case having a perfectly fair coin now seems rather unlikely.)

Your prior might depend not only on your general beliefs about coins but also on the context. If you pulled the coin out of your own pocket, you might be virtually certain that it's fair; if your magician friend pulled it out of his, your prior might put more weight on double-headed coins.

In any case, it's easy to come up with reasonable priors that (i) put a large probability on the coin being fair and (ii) lead your posterior to be quite similar, even after observing 220 heads. You'd then conclude that the coin was very likely to be fair, despite observing an outcome 2 SDs from the mean.

In fact, you could also construct examples where observing 220 heads in 400 tosses makes your posterior put more weight on the coin being fair, for example if all unfair coins have a probability of heads in $\{0, 1\}$.

Can anyone shed some light on this for me?

---

After writing this question I remembered that I had heard about this general situation before -- isn't it Lindley's "paradox"?

Whuber put a very interesting link in the comments: You Can Load a Die, But You Can't Bias a Coin. From page 3:

It does not make sense to say that the coin has a probability p of heads, because it can be completely determined by the manner in which it is tossed— unless it is tossed high in the air with a rapid spin and caught in the air with no bouncing, in which case p = 1/2.

Pretty cool! This ties into my question in an interesting way: suppose we know that the coin is being "tossed high in the air with a rapid spin and caught in the air with no bouncing." Then we definitely shouldn't reject the hypothesis that the coin is fair (where "fair" now means "having p=1/2 when tossed in the way described above"), because we effectively have a prior that puts all probability on the coin being fair. Maybe that justifies to some degree why I am uncomfortable rejecting the null after 220 heads are observed.

Answer 1

The standard Bayesian way to solve this problem (without Normal approximations) is to explicitly state your prior, combine it with your likelihood, which is Beta-distributed. Then integrate your posterior around 50%, say two standard deviations or from 49%–51% or whatever you like.

If your prior belief is continuous on [0,1] — e.g. Beta(100,100) (this one puts a lot of mass on roughly fair coins) — then the probability that the coin is fair is zero since the likelihood is also continuous [0, 1].

Even if the probability that the coin is fair is zero, you can usually answer whatever question you were going to answer with the posterior over the bias. For example, what is the casino edge given the posterior distribution over coin probabilities.

Answer 2

Let's say for the Bernoulli Distribution, in this case the toss of a coin.

Clearly this is a binomial distribution $B(n=400,p=0.5)$, and it is indeed close to $N(\mu=200,\sigma^2=100)$.

Obviously the interviewer is asking for the result of $k$ with $95\%$ confidence interval with $B(n=400,p=0.5)$, or the $p$-value of $B(n=400,p=0.5,k=220)$.

In the Bayesian approach, your prior is that $p=0.5$ instead of $\pi(p=0.5)=0.5$ and $\pi(p\neq0.5)=0.5$

Let's use some other more fair prior that $\pi(0.49\leq p\leq0.51)=0.9$ and $\pi(p<0.49 \cup p>0.51)=0.1$. We assume $p$ has uniform distribution within each interval.

We then can calculate the posterior $P(0.49\leq p\leq0.51|k=220)$.

Or highly likely the prior is a normal distribution $p$ ~ $N(\mu=0.5,\sigma^2=0.25)$, or we may assume much smaller variance such as $\sigma^2=0.1$.

Then we calculate the posterior distribution of $p$ as $f(p|k=220)$.

---

My reputation is not enough for me to write comment under the Question. Instead I'm gonna write something here regarding You Can't Bias a Coin. @Adrian

Here's what we have

1. The experiment result $B(n=400,k=220,p=\theta)$
2. The theoretical and experiment study You Can't Bias a Coin

Here's our Hypothesis

$H_0:$ The coin is fair, or $\hat\theta=0.5$

$H_1$: The experiment data is wrongly recorded

Here's our result

1. Based on the paper You Can Load a Die, But You Can't Bias a Coin, we accept hypothesis $H_0$.
2. Based on the experiment result that the difference is twice the standard deviation, we roughly have 95% confidence level to accept hypothesis $H_1$, that the experiment study is wrongly recorded.

Since the $p$-value for hypothesis test to reject either $H_0$ or $H_1$ is roughly below 5%, we must accept them both. Or we must reject them both.

Otherwise we create double standard for hypothesis testing here. We cannot accept the Hypothesis that the toss of coin is fair and the experiment data is correctly recorded.

---

It does not make sense to say that the coin has a probability p of heads

We have experiment result to back up this hypothesis.

If the experiment is repeated n times, is it possible that we have the prior of $p$ for the coin toss as $N(\mu=0.5,\sigma^2)$ when n is considerably large?

If that is acceptable, we can then estimate $\sigma^s$ with 95% CI based on method of maximum likelihood.