Check whether a coin is fair

Question

An interview problem is like the following:

Given a coin you don’t know it’s fair or unfair. Throw it 6 times and get 1 tail and 5 head. Determine whether it’s fair or not. What’s your confidence value?

I came out the following solution:

$H_0:$ the coin is fair $H_a:$ the coin is unfair

$X$: is the number of heads

Rejection region: $|X - 3| > 2$, i.e., $X = 0,1,5,6$

Significance level alpha:

$\alpha = P(\text{reject }H_0 \mid H_0 \text{ is true})$ $=P(X=0,1,5,6 \mid H_0 \text{ is true})$ $= (\binom{6}{0}+\binom{6}{1}+\binom{6}{5}+\binom{6}{6})*(1/2)^6$ $= (1+6+6+1)*(0.5^6) = 0.21875$

because $\alpha > 0.05$, we do not have enough evidence to reject $H_0$, and we accept $H_0$, so the coin is fair

Is the above test a good one? And I did not know how to calculate the confidence value?

Answer 1

[I think I'd start by asking for a whiteboard, markers -- and an eraser, because one boardful isn't enough to explain everything wrong with the question.]

I'm going to answer this question by rejecting its premises.

1. The "coin" itself is just a coin; by itself it doesn't do anything, and so it cannot be fair or not-fair. What we're talking about is the process of tossing a particular coin in some fashion -- that can be discussed in terms of whether it's fair or not.

2. Data can't show you that a coin-tossing process applied to some coin is exactly fair. Sometimes it can show you that your coin-tossing-process on a given coin is inconsistent with fairness, but failure to identify any inconsistency with fairness doesn't imply fairness (failure to reject is because your sample size is small, not because the coin is actually fair).

   [e.g. Consider it in terms of a confidence interval for P(head), the fact that $\frac12$ is in the CI doesn't mean that P(head)=$\frac12$, since there are always other values - distinct from $\frac12$ - in there too. Or think in terms of power: on the experiment given in the interview question - 6 tosses - what's the probability that you'd reject as unfair the case where the tossing process applied to a particular coin had $p(\text{head})=0.51$ at some typical significance level? That's clearly an unfair coin, but you'll reject barely more often than your type I error rate, and a large fraction of those rejections in a two tailed test would be "in the wrong tail"!]

3. No coin-tossing process on a given coin will be perfectly fair. (For example, changing the side facing up slightly alters the chances associated with the resulting face on the toss, as experiments run by Persi Diaconis have shown.)

   Could the coin be close to fair? Possibly; it may even be possible to get very close to fair. Exactly fair? No, it's not possible in practice. But then to discuss whether it's "close to fair" we'd have to define what we mean by 'close'. [If we were to give some usable definition, while some people might suggest some form of equivalence test, or perhaps considering whether some CI lay entirely inside some "close to fair" bounds, I'd be inclined toward a Bayesian approach to deciding whether the coin is sufficiently close to fair. Note that with the tiny sample size mentioned, the data are quite consistent with p(head) so far from $\frac{1}{2}$ that this exercise on that data would not conclude "close to fair" on any of the three mentioned approaches.]

So:

Given a coin you don’t know it’s fair or unfair.

Yes, actually, I do. In fact I don't even need to see data. It's not fair.

Throw it 6 times and get 1 tail and 5 heads. Determine whether it’s fair or not.

I really don't care what the data are. It makes no difference to my answer, since the data could not possibly demonstrate fairness, even if fairness were a realistically possible state to be in.

What’s your confidence value?

100% (in a sense similar to almost surely)

(In any case, even if there were a way to do this statistically I don't know of any statistical procedure that gives anything I'd agree to call "confidence values", so I also reject the form of that question. What does that term even mean? If I were asked a question phrased that way in an interview, I'd have serious concerns about working there, because it seems to suggest the people conducting the interview don't really understand what they're even asking - and that suggests either nobody there knows this stuff, or they don't care enough about this position to make sure the interview is being conducted by someone who does. Either way, it would certainly influence my willingness to work there.)

---

Forgetting everything I just said for the moment, some comments on your hypothesis test:

Your process for a hypothesis test is wrong.

1. Why do you compare your significance level with 0.05? You've chosen a significance level of 0.21 (which I have no objection to in this experiment, the sample size is so low you only have 3% or 21% and $\alpha$=3% will be too low-powered to be much use) -- 0.05 doesn't relate to anything here.

2. Do you see that in your test when it came time to reject or not reject, you made no reference at all to the sample statistic (5 heads)? Indeed you ignored your rejection rule.

3. The rejection rule you stated algebraically $|X-3|>2$ is inconsistent with the rejection region you mentioned ($0,1,5,6$).

That's a lot of errors in a few lines! If I was involved in such an interview**, I might forgive the error with the rejection rule as something one could overlook under interview pressure, but the first two errors would suggest some fundamental problems.

** leaving aside that I'd never ask such a poor question, nor would I likely care enough about hypothesis testing to even think to ask a question about it.

Answer 2

I am not going to attempt to provide final answers to your question; I believe the topic is more than addressed after the comprehensive response given by Glen. However, and apropos of his comment about a Bayesian approach, I'd like to post some illustrations about the way our preconceptions about the "fairness" of the coin (or the experiment in general) affects the posterior probability density, i.e. the $p\,(\theta\,\vert\,\text{Data})$, where $\theta$ stands for the probability of heads in the coin toss.

Luckily, we have a conjugate prior distribution for the binomial case that occupies us - the beta distribution, facilitating the calculation of the posterior distribution.

First scenario - The Fair-Minded Player:

We walk into the game (not a very exciting game, but still...), and we have absolutely no reason to assume that there is foul play going on. Things being by nature less than perfect, we have it in our mind that the coin is fair-ish. In other words, we think that the probability of heads, $\theta$, falls somewhere around $\frac{1}{2}$. Later, the unexpected single tail out of $6$ tosses, will force us to move the posterior probability of $\theta$ to the left (the arrows indicate the influence of the data on the prior distribution):

Second Scenario - The Shrewd Player:

We strongly suspect from insider's leaked information that the game is markedly biased towards tails, and we not only are about to make a killing, but also in need to further reinforce our conviction after the first round, doubling down our bet:

Third Scenario - Losing Your Shirt:

We've never played before, but we have read a manual, and we feel ready. All signs clearly indicate that the coin is markedly biased towards $heads$, a mistake that we will soon start to correct at a high $\ $\$ cost:

Fourth Scenario - No Idea Whatsoever:

It's a good thing that the $\beta(1,1)$ distribution turns into a $U\,(0,1)$ to address this scenario, where only the likelihood will influence the -posterior probability of $\theta$. As brought up to my attention, a Jeffreys prior is close and possibly more correct:

So I hope this provides a bit of a light-hearted visual depiction of our approach to estimating the chances of this game being rigged, perhaps encapsulating more of a real scenario than calculations of the type pbinom(1, 6, 0.5). If you want the code in R, and the credits to a great video with Matlab illustrations, I posted it here.

Answer 3

I'm thinking using chi square to measure the statistical difference between categorical variables.

Null hypothesis: half of the coins you tossed are heads and half are tails. Alternative hypothesis: opposite to the above

Then you calculate the chi square using this formula sum((f0-fe)^2/fe) where f0 is your statistic or point estimate and fe is the expected value. And then you compare this value with critical chi square value given from table to determine if you reject null hypothesis.