I have recently realized that I can estimate coin fairness, making two observations, either incrementally, through bayesian updating or immediately. I have heard that before but do not see the full picture to understand why does that work, why results are the same either way.

Unfair coin is considered which always gives heads, so two heads obsevation with priors .5/.5 for the coin being fair/unfair should give posteriors

$$P(fair| hh),P(unfair| hh) = {P(hh|f) \cdot P(f), P(hh|u) \cdot P(u) \over P(hh|f) \cdot P(f) + P(hh|u) \cdot P(u) } = \\= {.25 \cdot .5, 1 \cdot .5\over .25 \cdot .5 + 1 \cdot .5} = {1, 4\over 1+4} = {1\over 5},{4\over 5}$$.

The same can be obtained incrementally, first landing first heads

$$P(fair| h),P(unfair| h) = {P(h|f) \cdot P(f), P(h|u) \cdot P(u) \over P(h|f) \cdot P(f) + P(h|u) \cdot P(u)} = \\ = {.5 \cdot .5, 1 \cdot .5\over .5 \cdot .5 + 1 \cdot .5} = {1, 2\over 1+2} = {1\over 3},{2\over 3}$$

and flipping heads next time with posteriors becoming new priors, $P(f), P(u) = .5,.5 \to 1/3, 2/3$

$$P(fair| hh),P(unfair| hh) = {.5 \cdot 1/3, 1 \cdot 2/3\over .5 \cdot 1/3 + 1 \cdot 2/3} = {1, 4\over 1+4} = {1\over 5},{4\over 5}$$.

Results match. Multiple observations section basically says that

$$P(h^n|H) =P(h|H)^n$$

which seems to be true because, for two heads, we have $P(hh|f),P(hh|uf) = P(h|f)^2,P(h|u)^2 = .25 \text{ and } 1$ and we used them in the first equation to compute the posteriors after double throw. But, it is diffucult to see why iterative solution gives the same result.

If you let me, I denote $P(h|f)=a$ and $P(h|u)=b$. I also denote n-th prior with $p_n = p_{na}, p_{nb}$ because it is a pair of probabilities, one for fair coin and the other is expectation of unfair. Now, I can rewrite the above incremental equations

$$p_2 = {a\cdot p_{1a}, bp_{1b}\over a\cdot p_{1a}+ bp_{1b}}$$ $$p_3 = {a\cdot p_{2a}, bp_{2b}\over a\cdot p_{2a}+ bp_{2b}}$$

I do not see however how the latter is equal the posterior computation for the immediate double heads observation

$$p_3 = {a^2\cdot p_{1a}, b^2p_{1b}\over a^2\cdot p_{1a}+ b^2p_{1b}}$$

Should I resort to Ω₁ = Ω₀⁢ · LR formula, which stands for $$P(H₁|D)/P(H₂|D) = Ω₁ = Ω₀⁢ · LR = P(H₁)/P(H₂)⁢ · P(D|H₁) / P(D|H₂),$$ says that the only thing that matters in Bayesian is the ratio of probabilities and can ealsily give me n-th posterior ratio $Ω_n = Ω₀· LRⁿ$?


1 Answer 1


Wait, it is not a big deal to demonstrate. Let's take

$$p_{3a} = {a\cdot p_{2a}\over a\cdot p_{2a}+ bp_{2b}} = {a\cdot {a\cdot p_{1a}\over a\cdot p_{1a}+ bp_{1b}}\over a\cdot {a\cdot p_{1a}\over a\cdot p_{1a}+ bp_{1b}}+ b{a\cdot p_{1a}\over a\cdot p_{1a}+ bp_{1b}}} = {{a^2\cdot p_{1a}}\over {a^2\cdot p_{1a}}+ {b^2\cdot p_{1b}}}$$

Similarly, $$p_{3b} = {{b^2 p_{1b}}\over {a^2p_{1a}}+ {b^2p_{1b}}}$$

which makes

$$p_3 = {a^2 p_{1a}, b^2 p_{1b} \over {a^2p_{1a}}+ {b^2p_{1b}}}$$

as expected from the double heads immediate observation!

You see, the normalization factor from the previous iterative update cancels out and is replaced by new normalization factor at every next update. It is easy to understand why this happening. We have got new priors in the iteration by normalizing the joint probabilities P(Evidence and Prior) with that factor. On the next iteration, every P(Evidence and Prior) has this factor in it and we use new normalizing factor, which is the sum of the joint probabilities, which means that the sum also contains it. So, both nominator and denominator contain this factor and we, thus, can just skip all the intermediate normalizations and normalize only once in the end, as Allison Dawny put it when offers to do all the bayesian updates with denominator ignored.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.