Paradoxical results when performing simple Bayesian analysis. Need help explain results

Question

Recently, I ran into a case where I used two different methods to perform parameter estimation, and ended up with contradictory results. I am going to use a very simple example to illustrate the problem: estimating the parameter of a coin. I'll present two solutions to the same problem, and they seem to produce contradicting results.

Problem: Decide whether a coin is fair after observing 2 heads and 3 tails.

Definition and notation: Let θ be the probability of getting a head. A coin is fair if $θ\in(0.49, 0.51)$. In the following, I'll use notation $Data$ to denote the observation of $(H=2, T=3)$.

Assumption: Assume a uniform prior $P(θ)\sim Uniform(0,1)$ or equivalently $P(θ) \sim Beta[1,1]$

Hypothesis:

• H0: The coin is fair, $θ\in(0.49, 0.51)$.
• H1: The coin is not fair, $θ\in[0, 0.49]\cup[0.51, 1]$.

Note that H0 and H1 are mutually exclusive and they cover the whole [0, 1] range.

I have two solutions in which I cannot find mistakes, but they produced contradictory conclusions.

Solution 1: Bayesian analysis of P(θ|Data) $$ P(θ|Data) = \frac{P(Data|θ) * P(θ)}{P(Data)} = \frac{P(Data|θ) * P(θ)}{\int_0^1 P(Data|θ) * P(θ)\,dθ} $$

Let's pluging Data = (H=2, T=3), and P(θ)=1 for θ∈[0, 1].

Note that P(Data|θ) follows a Binomial distribution. $$ P(Data|θ) = θ^2 * (1-θ)^3 $$ So: $$ P(θ|Data) = \frac{P(Data|θ) * P(θ)}{\int_0^1 P(Data|θ) * P(θ)\,dθ} = \frac{θ^2 * (1-θ)^3 * 1.0}{\int_0^1 θ^2 * (1-θ)^3 * 1.0\,dθ} $$ We can paste the above expression into wolframalpha to verify that the above simplifies to: $$ P(θ|Data) = 60 (1 - θ)^3 θ^2 $$ We can also verify that this is a proper pdf since it integrates to 1 on [0,1].

Now we can compute P(0.49<θ<0.51|Data) by integrating P(θ|Data) over [0.49, 0.51]: $$ \int_{0.49}^{0.51} P(θ|Data)\,dθ = \int_{0.49}^{0.51} 60*(1 - θ)^3*θ^2\, dθ = 0.03749 $$ So $P(H0|Data) = 0.03749$ and $P(H1|Data) = 1 - P(H0 | Data) = 0.96251$

The conclusion is that H1 is much more likely to be correct: the coin is likely unfair.

Solution 2: Analysis of data likelihood P(Data|H0) and P(Data|H1)

Let's look at H0 first: θ∈(0.49, 0.51). Assuming that H0 is true, properly normalizing the uniform prior distribution gives P(θ)=50 for θ∈(0.49, 0.51). $$ P(Data|H0, θ) = θ^2 * (1-θ)^3 $$ Marginalizing over θ (computing expectation over θ): $$ \begin{align} P(Data|H0) & \\&= \int_{0.49}^{0.51} θ^2 * (1-θ)^3 * P(θ)\, dθ \\&= \int_{0.49}^{0.51} θ^2 * (1-θ)^3 * 50 \, dθ \\&= 0.0312417 \end{align} $$ (One quick way to get an approximation to this integral is to just calculate $P(Data|H0, θ=0.5)$, which gives 1/32=0.03125. This also verifies the math above.)

Now we assume that H1 is true: θ∈[0, 0.49]∪[0.51, 1]. Assuming that H1 is true instead, properly normalizing the uniform prior distribution gives P(θ)=1/0.98 for θ∈[0, 0.49]∪[0.51, 1]. Using exactly the same reasoning: $$ P(Data|H1, θ) = θ^2 * (1-θ)^3 $$ Marginalizing over θ (computing expectation over θ): $$ \begin{align} P(Data|H1)& \\&= \int_{0}^{0.49} θ^2 * (1-θ)^3 * P(θ)\, dθ + \int_{0.51}^{1} θ^2 * (1-θ)^3 * P(θ)\, dθ \\&= \int_{0}^{0.49} θ^2 * (1-θ)^3 * 1/0.98\, dθ + \int_{0.51}^{1} θ^2 * (1-θ)^3 * 1/0.98\, dθ \\&= 0.0163692 \end{align} $$ With $P(Data|H0) = 0.0312417$ and $P(Data|H1) = 0.0163692$, the conclusion is that the coin is more likely fair.

I've verified my reasoning and math above for both solution, and couldn't spot any errors. But they clearly produced contradictory conclusions. Can anyone help me explain? Thank you.

Answer 1

There is no paradox.

The posterior odds in favor of H0 relative to H1 equals the prior odds times the Bayes factor. When you change the prior odds you change to posterior odds.

In "Solution 1" the prior odds in favor of H0 are 2/98 = 1/49. In "Solution 2" the prior odds are 1/1. Therefore, the posterior odds in favor of H0 according to "Solution 2" is 49 times that of the posterior odds in favor of H0 according to "Solution 1": $$ \underbrace{0.0312417\div 0.0163692}_{\text{Soln 2 posterior odds}} = 49 \times \underbrace{0.03749\div 0.96251}_{\text{Soln 1 posterior odds}}. $$

(BTW, I did not check any of the work. I simply went straight to the posterior odds ratios on the theory that everything up to that point was done correctly.)

Answer 2

Intuitively:

• Solution 2 gives a good result. An outcome of 2H/3T seems plausible when tossing a fair coin.

• Assume an experiment of 0 coin tosses. You won't be able to tell anything about the fairness of the coin - the posterior is the same as the prior: the uniform distribution. However, approach 1 will still tell you that the coin is not fair because you integrate over a much smaller range under H0.

It appears, the problem with solution 1 is that it does not take the unequal ranges covered by the hypotheses into account. Solution 2 does that when you normalize the priors.

I have to stop for now. If I find the time, I will add some formal considerations later.