12
$\begingroup$

I was asked this question with $(n, k) = (400, 220)$ in an interview. Is there a "correct" answer?

Assume the tosses are i.i.d. and the probability of heads is $p=0.5$. The distribution of the number of heads in 400 tosses should then be close to Normal(200, 10^2), so that 220 heads is 2 standard deviations away from the mean. The probability of observing such an outcome (i.e. more 2 SDs away from the mean in either direction) is slightly less than 5%.

The interviewer told me, essentially, "if I observe something >= 2 SDs from the mean, I conclude that something else is going on. I would bet against the coin being fair." That's reasonable -- after all, that's what most hypothesis tests do. But is that the end of the story? For the interviewer that seemed to be the "correct" answer. What I'm asking here is whether some nuance is justified.

I couldn't help but point out that deciding that the coin is not fair is a bizarre conclusion in this coin-tossing context. Am I right to say that? I'll try and explain below.

First of all, I -- and I would assume most people as well -- have a strong prior about coins: they're very likely to be fair. Of course that depends on what we mean by fair -- one possibility would be to define "fair" as "having a probability of heads 'close' to 0.5, say between 0.49 and 0.51."

(You could also define 'fair' as meaning that the probability of heads is exactly 0.50, in which case having a perfectly fair coin now seems rather unlikely.)

Your prior might depend not only on your general beliefs about coins but also on the context. If you pulled the coin out of your own pocket, you might be virtually certain that it's fair; if your magician friend pulled it out of his, your prior might put more weight on double-headed coins.

In any case, it's easy to come up with reasonable priors that (i) put a large probability on the coin being fair and (ii) lead your posterior to be quite similar, even after observing 220 heads. You'd then conclude that the coin was very likely to be fair, despite observing an outcome 2 SDs from the mean.

In fact, you could also construct examples where observing 220 heads in 400 tosses makes your posterior put more weight on the coin being fair, for example if all unfair coins have a probability of heads in $\{0, 1\}$.

Can anyone shed some light on this for me?


After writing this question I remembered that I had heard about this general situation before -- isn't it Lindley's "paradox"?

Whuber put a very interesting link in the comments: You Can Load a Die, But You Can't Bias a Coin. From page 3:

It does not make sense to say that the coin has a probability p of heads, because it can be completely determined by the manner in which it is tossed— unless it is tossed high in the air with a rapid spin and caught in the air with no bouncing, in which case p = 1/2.

Pretty cool! This ties into my question in an interesting way: suppose we know that the coin is being "tossed high in the air with a rapid spin and caught in the air with no bouncing." Then we definitely shouldn't reject the hypothesis that the coin is fair (where "fair" now means "having p=1/2 when tossed in the way described above"), because we effectively have a prior that puts all probability on the coin being fair. Maybe that justifies to some degree why I am uncomfortable rejecting the null after 220 heads are observed.

$\endgroup$
  • $\begingroup$ Would any part of your question change if you were to interpret the "coin" as a metaphor for some binary process about which you had no prior knowledge? $\endgroup$ – whuber Apr 29 '14 at 19:17
  • 1
    $\begingroup$ @whuber That's a good question. I think in that case I'd be much more willing to go with "reject when p <= 0.05", although I'm not quite sure of how to justify that to myself. $\endgroup$ – Adrian Apr 29 '14 at 19:24
  • $\begingroup$ Another issue that bothers me is that the person asking the question was interested in the hypothesis that p=0.50 exactly. But if you think of p being continuously distributed, that's going to have probability zero, regardless of what you observe. It strikes me as much more meaningful to make statements about p belonging to some interval. This would be an issue in the situation where I had no prior knowledge and decided to use a uniform prior, for example. $\endgroup$ – Adrian Apr 29 '14 at 19:26
  • 3
    $\begingroup$ It makes sense. The coin-focused question is a bit distracting, though, because answers to such questions typically devolve into discussions of the physics (and sleight-of-hand) of coin flipping. You might be shocked at how contrary the actual situation can be to your strong priors, depending on how the coin is flipped. "It does not make sense to say that the coin has a probability $p$ of heads". $\endgroup$ – whuber Apr 29 '14 at 19:56
  • 1
    $\begingroup$ @Adrian DJC MacKay discusses this exact problem (with n=250, k=140) in his free textbook at this link: inference.phy.cam.ac.uk/itprnn/book.pdf (p63.) It might be interesting to read what he says. He reaches a similar conclusion to you. $\endgroup$ – Flounderer Jul 8 '14 at 0:33
3
$\begingroup$

The standard Bayesian way to solve this problem (without Normal approximations) is to explicitly state your prior, combine it with your likelihood, which is Beta-distributed. Then integrate your posterior around 50%, say two standard deviations or from 49%–51% or whatever you like.

If your prior belief is continuous on [0,1] — e.g. Beta(100,100) (this one puts a lot of mass on roughly fair coins) — then the probability that the coin is fair is zero since the likelihood is also continuous [0, 1].

Even if the probability that the coin is fair is zero, you can usually answer whatever question you were going to answer with the posterior over the bias. For example, what is the casino edge given the posterior distribution over coin probabilities.

$\endgroup$
2
$\begingroup$

Let's say for the Bernoulli Distribution, in this case the toss of a coin.

Clearly this is a binomial distribution $B(n=400,p=0.5)$, and it is indeed close to $N(\mu=200,\sigma^2=100)$.

Obviously the interviewer is asking for the result of $k$ with $95\%$ confidence interval with $B(n=400,p=0.5)$, or the $p$-value of $B(n=400,p=0.5,k=220)$.

In the Bayesian approach, your prior is that $p=0.5$ instead of $\pi(p=0.5)=0.5$ and $\pi(p\neq0.5)=0.5$

Let's use some other more fair prior that $\pi(0.49\leq p\leq0.51)=0.9$ and $\pi(p<0.49 \cup p>0.51)=0.1$. We assume $p$ has uniform distribution within each interval.

We then can calculate the posterior $P(0.49\leq p\leq0.51|k=220)$.

Or highly likely the prior is a normal distribution $p$ ~ $N(\mu=0.5,\sigma^2=0.25)$, or we may assume much smaller variance such as $\sigma^2=0.1$.

Then we calculate the posterior distribution of $p$ as $f(p|k=220)$.


My reputation is not enough for me to write comment under the Question. Instead I'm gonna write something here regarding You Can't Bias a Coin. @Adrian

Here's what we have

  1. The experiment result $B(n=400,k=220,p=\theta)$
  2. The theoretical and experiment study You Can't Bias a Coin

Here's our Hypothesis

$H_0:$ The coin is fair, or $\hat\theta=0.5$

$H_1$: The experiment data is wrongly recorded

Here's our result

  1. Based on the paper You Can Load a Die, But You Can't Bias a Coin, we accept hypothesis $H_0$.
  2. Based on the experiment result that the difference is twice the standard deviation, we roughly have 95% confidence level to accept hypothesis $H_1$, that the experiment study is wrongly recorded.

Since the $p$-value for hypothesis test to reject either $H_0$ or $H_1$ is roughly below 5%, we must accept them both. Or we must reject them both.

Otherwise we create double standard for hypothesis testing here. We cannot accept the Hypothesis that the toss of coin is fair and the experiment data is correctly recorded.


It does not make sense to say that the coin has a probability p of heads

We have experiment result to back up this hypothesis.

If the experiment is repeated n times, is it possible that we have the prior of $p$ for the coin toss as $N(\mu=0.5,\sigma^2)$ when n is considerably large?

If that is acceptable, we can then estimate $\sigma^s$ with 95% CI based on method of maximum likelihood.

$\endgroup$
  • 1
    $\begingroup$ Thank you Zhang. One tiny nit: if you wanted to use the Normal distribution for your prior over the probability of heads, I'd say you should truncate it so that p lies in [0, 1]. $\endgroup$ – Adrian Apr 29 '14 at 19:07
  • $\begingroup$ Of course there are many reasonable prior distributions and corresponding posteriors. The real point of my question is more general: deciding that the coin is not fair seems to me to be a bizarre conclusion in this coin-tossing context. What do you think about that -- and why? $\endgroup$ – Adrian Apr 29 '14 at 19:09
  • $\begingroup$ A convenient prior here would be the Beta distribution, since it's conjugate to the Binomial likelihood. But again, the real thrust of my question is more general than the specific prior. $\endgroup$ – Adrian Apr 29 '14 at 19:11
  • $\begingroup$ I think the reason is that you give too much probability to the specific prior $\pi(p=0.5)$. I think if you change the prior, namely a simply uniform distribution $p \sim U(0,1)$, and we construct a 95% confidence interval for $E(p) \sim f(p|k=220)$, I think we will find the result to be much convincing, with $p=0.5$ not lying in the 95% CI of $E(p)$. And we easily accept the hypothesis that the coin is not fair. Especially in this case, you won't find deciding the coin as not fair to be a bizarre conclusion. $\endgroup$ – Zhang Tschao Apr 30 '14 at 2:17
  • $\begingroup$ @user777 The normal distribution appears twice in Zhang's response, first as an approximation to the binomial (great), and second as a prior for the probability of heads (when he says "the prior is a normal distribution p ~ N"). Zhang -- your edit about the Null being "the coin is fair and the data were correctly recorded" is interesting, thank you for posting it. $\endgroup$ – Adrian Apr 30 '14 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.