# Testing if a coin is fair using Bayesian statistics

Suppose we have a coin and want to decide whether it's fair. We assume that the a priori probability of a coin being fair is 1/2. However, we can't yet calculate the probability of the outcome of a coin flip, because to do so would require data we do not have. How would we estimate the probability that this coin is fair?

Idea: We can conduct an experiment where in we flip the coin 1000 times. We count the number of successes (either heads or tails), and calculate the probability that this number of successes occurs in a fair coin*. We'll call this Pfair.

We then estimate the probability of this number of successes occurring in this coin* by assuming that the probability of a single success = the number of successes / 1000. We'll call the resulting binomial probability Pcoin. Then, we can estimate that P(success) = Pfair * P(fair coin) + Pcoin * P(unfair coin) = (Pfair + Pcoin) / 2.

*= Alternatively, we can calculate the probabilities that the number of successes is either less than or greater than the expected/actual number of successes.

Edit: What I'm after is more of a model of the likelihood that a given model is true. To do this, I would need the a priori probability of observing the data. Suppose I don't know that the probability that I pick up a random coin and get heads is 1/2. How can I apply Bayes' rule?

• Whether or not you have data, you have beliefs about coin fairness (e.g., the coin is fair unless data suggest otherwise, or the coin is unfair, giving heads probability P, or there is a distribution of possible degrees of fairness representing how uncertain you are about the (un)fairness of the coin, etc.). – Alexis Aug 26 '18 at 17:03

Bayesian hypothesis testing is usually done by formulating a model that decomposes the prior into the null and alternative cases, which leads to a particular form for Bayes factor. For an arbitrary prior probability for the null hypothesis we can update to find the posterior probability of the null hypothesis using a simple equation involving Bayes factor. This leads to a simple graph of the prior-to-posterior probability mapping, which is a bit like an AUC curve. Here is a general model form for your problem, plus the more specific uniform-prior model.

General Bayesian model: Suppose we observe coin tosses yielding the indicator variables $$x_1, x_2, ..., x_n \sim \text{IID Bern}(\theta)$$ where one indicates heads and zero indicates tails. We observe $$s = \sum_{i=1}^n x_i$$ heads in $$n$$ coin tosses, and we want to use this data to test the hypotheses:

$$H_0: \theta = \tfrac{1}{2} \quad \quad \quad H_A: \theta \neq \tfrac{1}{2}.$$

Without any loss of generality, let $$\delta \equiv \mathbb{P}(H_0)$$ be the prior probability of the null hypothesis and let $$\pi_A(\theta) = p(\theta|H_A)$$ be the conditional prior for $$\theta$$ under the alternative hypothesis. For this arbitrary prior we can express Bayes factor as:

\begin{aligned} BF(n,s) \equiv \frac{p(\mathbf{x}|H_A) }{p(\mathbf{x}|H_0)} &= \frac{\int_0^1 \theta^s (1-\theta)^{n-s} \pi_A(\theta) d\theta}{(\tfrac{1}{2})^n} \\[6pt] &= 2^n \int_0^1 \theta^s (1-\theta)^{n-s} \pi_A(\theta) d\theta. \\[6pt] \end{aligned}

We then have posterior:

\begin{aligned} \mathbb{P}(H_0|\mathbf{x}) = \mathbb{P}(H_0|s) &= \frac{p(\mathbf{x}|H_0) \mathbb{P}(H_0)}{p(\mathbf{x}|H_0) \mathbb{P}(H_0) + p(\mathbf{x}|H_A) \mathbb{P}(H_A)} \\[6pt] &= \frac{\mathbb{P}(H_0)}{\mathbb{P}(H_0) + BF(n,s) \mathbb{P}(H_A)} \\[6pt] &= \frac{\delta}{\delta + (1-\delta) BF(n,s)}. \\[6pt] \end{aligned}

Use of a particular conditional prior $$\pi_A$$ leads to different forms for the Bayes factor,

Testing with uniform prior under alternative: Suppose we take $$\pi_A(\theta) = \mathbb{I}(\theta \neq 1/2)$$ so that this conditional prior is uniform over the allowable parameter values. (We could take $$\pi_A(\theta) = 1$$ since changing the density at a single point has no effect; thus, we can be a bit fast-and-loose with the support.) Under this model the Bayes factor is:

$$BF(n,s) = 2^n \int_0^1 \theta^s (1-\theta)^{n-s} d\theta = 2^n \cdot \frac{\Gamma(s+1) \Gamma(n-s+1)}{\Gamma(n+2)}.$$

We can implement this model in R using the code below. In this code we give a function for the Bayes factor, and we generate the prior-posterior plot for some example data.

#Create function to calculate Bayes factor
BF <- function(n,s) { exp(n*log(2) + lbeta(s+1, n-s+1)) };

#Example with n = 1000 and s = 449
n <- 1000;
s <- 449;
B <- BF(n, s);

#Generate prior-posterior plot for of null hypothesis
library(ggplot2);

FIGURE <- ggplot(data.frame(delta = c(0, 1)), aes(delta)) +
stat_function(fun = function(delta) { delta/(delta + (1-delta)*B) },
colour = 'red', size = 1) +
ggtitle('Prior-Posterior Plot - Bayesian Hypothesis Test for Fair Coin') +
labs(subtitle = paste0('(Number of tosses = ', n,
', Number of heads = ', s, ')')) +
xlab('Prior Probability of Fair Coin') +
ylab('Posterior Probability of Fair Coin');

FIGURE;


• It’s worth mentioning, in view of Lindley’s paradox, that your inferences here are going to depend a lot on your prior. The uniform prior makes it really hard to reject when the true probability is close to, but not quite, 1/2, which is probably the case with an actual coin. – guy Aug 26 '18 at 20:14
• True, but it is inherently difficult to distinguish parameter values that are close together, and this naturally requires a lot of data. So I don't really see that as a drawback of the method; it is just a natural aspect of statistics. – Ben Aug 27 '18 at 0:37
• It's hard to distinguish points that are close together, sure. But Bayesian inference under a uniform prior diverges drastically from Frequentist inference in this particular case. I can decrease the Bayes factor in favor of the alternative by a factor of 5 just by consider a Uniform(.4,.6) prior under the alternative rather than a Uniform(0,1); this applies in settings where the Uniform(.4,.6) prior and Uniform(0,1) prior result in essentially the same inference about $\pi$ when you don't include a point mass at $1/2$. – guy Aug 27 '18 at 2:11
• Let me see if I can dumb down your answer a bit: We basically comsider not just the probability of experiencing a result for the null hypothesis, we alsp consider the probability of experiencing the result we got given a whole range of heads probabilities from 0 to 1 – moonman239 Sep 15 '18 at 2:59
• @moonman: Yes, that is the essence of Bayesian analysis --- we have an unknown probability of heads, represented by a parameter $\theta$, and we give this a prior distribution and then determine the posterior from the data. In hypothesis testing this generally entails giving a distribution under specific values, not just the general alternative hypothesis. – Ben Sep 15 '18 at 3:39

The common approach for dealing with this kind of problem (in particular, where you have assumed a priori a fair coin) is to use an appropriate concentration inequality.

For this specific case, you would want to use the Hoeffding bound, and apply it to Bernoulli random variables. Wikipedia has an entry just on that. To summarize, basically, one can calculate a bound on the probability of what you observed.

For example, what is the probability, given our assumption of a fair coin, that we will have seen at least (or at most) $X$ heads over 1000 coin flips? They give a nice result there where they show that (if we normalize our results by the number of flips, $n$), that the probability of the sum / $n$ being off by more than $\sqrt\frac{\ln n}{n}$ of the correct probability is at most $\frac{2}{n^2}$. Of course, you can put in different values, trading off the accuracy against the confidence level.

For completeness, the normalized version of the formula would be $P\left[\left|\frac{H(n)}{n} - p\right| > \epsilon\right] < 2e^{-2\epsilon^2n}$, where $H(n)$ is the sum of the Bernoulli trials, and $p$ is the actual probability for $1$.

• OP is specifically asking for a Bayesian solution. This answer is based entirely on frequentist considerations. – guy Aug 26 '18 at 16:33
• In addition to @guy 's comment, even if you go the frequentist route, why not just use the Binomial distribution? – jbowman Aug 26 '18 at 16:49

To be fair, there isn't a good method of testing if a coin is "fair" in Bayesian methodology. You should either use Fisher's method of maximum likelihood or Pearson and Neyman's Frequentist method, depending on your goal. The method cited by MotiN is a rigorous version of ROPE, or region of practical equivalence. This creates a region around $\pi=\frac{1}{2}$ where no one could distinguish between $H_0:\pi=\frac{1}{2}$ and $H_A:\pi\ne\frac{1}{2}$.

This is not the same as testing to determine if $\pi=\frac{1}{2}$. It is used for that purpose in Bayesian methods, but Bayesian methods are very ill-suited for testing sharp null hypotheses. The issue comes from the fact that the number of points in the interval [0,1] is uncountable, while $\pi=\frac{1}{2}$ is a countable point. As such, the null hypothesis has zero measure and so has zero probability. $\Pr\left(\pi=\frac{1}{2}\right)=0$.

Likewise, you could use Bayes factors against any single point to get a relative probability, but not a point versus a region, which is what is required here.

If you really need to know if a coin is fair, which you cannot really know, then you should use one of the two classical methods. Use Fisher's method of maximum likelihood if you want the p-value to provide information regarding the weight of the evidence against, or use the Frequentist method if you want to know how you should behave. If the null is not rejected, you behave as if it is true. If the null is rejected, then you behave as if it is false.

There is no good Bayesian answer, but if you must use a Bayesian method, then ROPE is as close as you can get.

• I don't agree with this answer. Since when can Bayesians not build Bayes factors for testing composite alternatives? Even though [0,1] is uncountable, there is nothing stopping you from putting a point mass at 1/2; this part of your answer makes the presumption that we are limited to continuous priors. – guy Aug 26 '18 at 16:36
• @guy are you presuming then that the prior has holes in it? What would be your prior? The Lesbegue measure of $[0,.5)\cup(.5,1]=1$. The Lesbegue measure of $\pi=\frac{1}{2}$ is the compliment, zero. The Bayes factor is zero. – Dave Harris Aug 26 '18 at 17:14
• My prior comment should have read @guy are you presuming then that the prior has holes in it? What would be your prior? The Lesbegue measure of $\Pr(\pi)$ over $[0,.5)\cup(.5,1]=1$. The Lesbegue measure of $\pi=\frac{1}{2}$ is the compliment, zero. The Bayes factor is zero. – Dave Harris Aug 26 '18 at 17:22
• You are making a massive assumption in stating that the prior on $\pi$ must have a density with respect to Lebesgue measure. Instead, we can have a density with respect to the sum of Lebesgue measure and the Dirac mass at 1/2, which is the completely standard thing to do. – guy Aug 26 '18 at 19:57