# Bad coin factory: Do coins in a set tend to be unfair?

The question I'm trying to answer is (fundamentally) this: I have a bag of coins that I suspect are weighted, some towards heads, some towards tails. I toss each coin 4 times and record the outcomes (e.g., 3H1T). As a group, do the coins tend to be unfair?

I can't figure out what an appropriate test would be, though it seems like there ought to be one. Here are some relevant thoughts and options I've considered.

(1) Binomial test - Appropriate way to test EACH COIN's fairness, but (a) 4 tosses isn't enough for statistical significance ($\alpha$ = .05) at the level of the individual coin and (b) since I suspect different coins may be weighted in opposite directions, lumping all the data together would make these coins cancel each other out. (See similar comments on this question)

(2) Chi-square goodness-of-fit or multinomial test over counts - This will tell me if my observed counts for each outcome (4H0T, 3H1T, 2H2T...) differ from the expected counts (they do), but not how. It will return a high test statistic whether my coins are all magically fair (all 2H2T results) or if they are all weighted (all either 4H0T, or 0H4T). It also ignores the underlying binomial nature of the data.

(3) Regression seems like overkill for this data, and linear regression/linear mixed models wouldn't answer the right question anyway: coins with opposite weighting would cancel each other out.

For reference, my actual counts are as follows, of a total of 56 "coins". Unfortunately, since they're not real coins, and the experiment is over, I can't just go flip each one a few more times!

16 4H, 10 3H, 9 2H, 8 1H, 13 0H

• What exactly is the hypothesis you're trying to test? That at least one coin in your bag is biased? – dsaxton Jun 19 '17 at 20:23
• @dsaxton -- No, I suppose H0 is that the coins are fair, and H1 is that the coins are weighted. So this is a question about the group-level properties of the set: Do I have enough extreme outcomes that I should comfortable saying the coins from this factory are not fair? – clukyanenko Jun 19 '17 at 21:01
• Negation of $H_0$ is "one of the coins at least is weighted". Do you assume (in $H_1$) all coins are equally weighted (same probability for heads) or something like that ? – Benoit Sanchez Jun 19 '17 at 21:52
• yeah, this is the problem I guess. I'm not super-clear on what I mean by "weighted". Maybe what I'm asking is, assuming that the coins are all weighted to the same degree, but not necessarily the same direction (so perhaps coins are all either P(H) = .75 or P(H) = .25), what degree of weighting gets me the best fit for the observed data? 75/25, 1/0, 55/45...? And that doesn't sound like such a simple probability question at all, now does it? – clukyanenko Jun 19 '17 at 22:10
• Especially since I have no a priori reason to expect that the number of coins that tend toward H should be the same as the number that tend toward T. There are a priori reasons to think that coins should be unfair to the same degree, just not to predict how many in each direction. – clukyanenko Jun 19 '17 at 22:17

The Multinomial Test seems appropriate. You can do this through a significance test, or by using Bayesian inference with a Dirichlet prior. For a frequentist significance test, your null hypothesis is: $$\begin{split} P(4H) &= 1/16 \\ P(3H) &= 1/4 \\ P(2H) &= 3/8 \\ P(1H) &= 1/4 \\ P(0H) &= 1/16. \end{split}$$

That's the probability distribution of an outcome of four flips of a fair coin. What you're trying to find is if the distribution of "head count" outcomes represents the distribution given by a set of fair coins. While this won't answer the question of whether any one coin is unfair (which, for the reasons you stated in your question, you just cannot do) this does help answer whether your set of coins are all fair. If the probability of getting an outcome that's as likely or less likely than yours given that they're all fair coins, then that's a reason to reject the null hypothesis.

As mentioned in the article I linked, the Multinomial test is intractable for a large number of samples and categories. That's because, for $N$ samples and $k$ multinomial categories, there are $N + k - 1 \choose k - 1$ possible outcomes, and you have to figure out which outcomes, from that set, are less likely than the one you got. So an approximation scheme like the likelihood ratio test might be appropriate. In your case ($N=56$, $k=5$) there are 490 thousand possible outcomes. If you don't mind the computational load, you can brute force calculate the probability of all those outcomes and search for which ones are lower than, or equal to, the probability of your outcome. The sum of those probabilities is your $p$-value. Otherwise you can use the LR test as illustrated in the wiki article, which is simple enough.

• So I actually ran this test before posting the question, and my distribution is reliably different from the predicted distribution, calculated as you show. But this doesn't test the right hypothesis, does it? I'll reject the null even if my distribution is all 2H, right? That result would be surprising, maybe indicating some funny non-independence of the coin flips, but it wouldn't suggest that coins from this factory are likely to either be weighted towards heads or towards tails. – clukyanenko Jun 19 '17 at 21:04
• Every test has assumptions. Independence of different coin flips is one of the safest assumptions you can make, and under that assumption this test is the obvious choice. – Paul Jun 19 '17 at 21:22
• I mean, once you introduce non-independence of the flips, you're opening the door to just about anything. The coins could conspire to produce a perfect multinomial distribution despite being mostly unfair. – Paul Jun 19 '17 at 21:26
• @clukyanenko If we got something like your example (all 56 experiments landing in 2H) I would absolutely conclude that this is not the result of a series of independent flips of fair coins. It's just too unlikely to get that result given said assumption. I just wouldn't know what to conclude. Hypothesis tests only tell us what to reject, not what to accept. – Bridgeburners Jun 19 '17 at 21:48
• Right. The speculation about independence in my previous comment is irrelevant. My point is that a result of all 2H would be unexpected given a set of independent flips of fair coins, and would (rightly) result in the rejection of the null using a multinomial test. But crucially, a result of all 2H wouldn't suggest that the coins from this factory tend to be weighted. Which means that the multinomial test isn't asking quite the question I want. – clukyanenko Jun 19 '17 at 21:55

What is unknown is about the factory. You want to test a property of the factory (not the coins).

While a coin can be entirely described by a single number (the probability of "heads"), a factory is naturally described by a distribution : the distribution of the random variable $P$ that is the probability of heads for a coin produced by this factory. The parameter $\theta$ is thus a distribution (not necessarily continuous) over $[0;1]$.

Now you can't observe $P$ directly : $P$ is latent. You can only observe a variable $X$ whose distribution given $P$ is $B(4,P)$. First let's calculate the distribution of $X$ (given $\theta$) :

$$P_\theta(X=k)=E_\theta\left(\binom{4}{k}P^k(1-P)^{4-k}\right)=\int_0^1\binom{4}{k}p^k(1-p)^{4-k}d\theta(p)$$ for $k=0,1,2,3,4$

This distribution $d(\theta)$ is an element $[0;1]^5$. $d(\theta)$ is not always a binomial distribution. It is a binomial distribution when $\theta$ is a Dirac. $(0.5;0;0;0;0;0.5)$ is a possible value for $d(\theta)$ that is not binomial. Yet $X$ cannot have ANY possible distribution. For example, the distribution $(0,0,1,0,0)$ is not a possible value for $d$.

Let's look at the set of all possible distributions : $$D=\{d(\theta)/\theta \text{ is a probability distribution over [0;1]}\}$$

Finding $D$ is maybe difficult. I won't try to do it. But you will have to I guess. It is convex : image of the convex space of distributions by a linear map. I think it must be a kind of polyhedron (like a simplex).

Now the idea is to use $d\in D$ as the parameter instead of $\theta$. The null hypothesis is $d=(1/16;1/4;3/8;1/4;1/16)$.

You can calculate the likelihood of a vector of 56 independent variables having $d$ as a distribution (multinomial stuff). Then you can use a likelihood ratio test as suggested by Bridgeburner. But the range of the parameters won't be every possible distribution, only those in $D$. See : https://en.wikipedia.org/wiki/Likelihood-ratio_test

I think this solves the problem of rejecting if all values are 2 : the distribution $(0;0;1;0;0)$ is not an element of $D$.

The difficult point seems to find $D$ and to calculate the maximum of the likelihood over $D$.

• When you say that $(0;0;1;0;0)$ is not a possible value of $d$ are you imposing the assumption that flips are necessarily independent? Because if so then I think $D$ is simply a one-dimensional manifold because you can simply calculate it based on a single parameter $p \in [0,1]$ of the coin (in the same way you calculated the null hypothesis with the assumption $p=1/2$). If that's not what you meant, then why is $(0;0;1;0;0)$ not a possible value of $d$? – Bridgeburners Jun 20 '17 at 18:32
• On second reading I did misunderstand. $D$ is more complex than I made it out to be. Even if you impose the independent flip assumption, $D$ is only a one dimensional space if $\theta$ is a delta function, which isn't necessarily the case. (I was assuming $d$ is binomial.) – Bridgeburners Jun 20 '17 at 19:01
• Yes I assume independence of the flips (once the coin has been created by the factory). This is the binomial given $p$. $d$is the distribution given $\theta$ and not the distribution given $p$. That's rather subtle. It is a mixture of infinitely many binomials ($\theta$ describes the weights of each binomial). The binomials are all in $D$ (a binomial is a mixture of single binomial). But not only binomials eg 1/2 0 0 0 0 1/2 is id $D$. Hence $D$ is the convex envelope of the one dimensional manifold that are the binomials. – Benoit Sanchez Jun 20 '17 at 19:04
• I need to think this through a little more carefully, but this certainly feels more like the answer I wanted -- It's aimed at testing the underlying distribution of coins, not the coins themselves. For practical purposes though (this is a minor point in a paper I'm writing), I'll probably end up sticking with Bridgeburners' original solution. – clukyanenko Jun 20 '17 at 22:50