Where is wrong with my formulation of estimating the probability of a biased coin?

Question

After a year and a half I've realized that the question is very ill posed and the notations are confusing and misleading, I tried to fix it but it seems irredeemable. Actually it took myself half an hour to figure out what I was asking in the first place. So to have yourself a good time, I would suggest you close this page and take a break from whatever lead you here.

---

I represent a biased coin with a discrete distribution $p(\theta)$, where $p(\theta=h)=\pi$ is the probability of heads, and $p(\theta=t)=1-\pi$ the probability of tails in one toss.

I have a terrible eye sight, so I have $1/6$ probability of getting the wrong observation.

Suppose I have a prior $p(\theta=h)=3/4$. Then the expectation of observing heads in one toss is then $E(x)=\frac{3}{4}\times\frac{5}{6}+\frac{1}{4}\times\frac{1}{6}=2/3$.

And I toss this coin three times and observe $x=\{head, head, tail\}$.

I want to compute the posterior probability $p(\theta|x)\propto p(x|\theta)p(\theta)$.

So the likelihood function in this case is $p(x|\theta=h)=(5/6)^2(1/6)$, $p(x|\theta=t)=(1/6)^2(5/6)$, hence the posterior probability is
$p(\theta=h|x)=\frac{p(x|\theta=h)p(\theta=h)}{p(x)}=\frac{(5/6)^2(1/6)(2/3)}{norm}=10/11$ $p(\theta=t|x)=\frac{p(x|\theta=t)p(\theta=t)}{p(x)}=\frac{(1/6)^2(5/6)(1/3)}{norm}=1/11$.

It seems that the observation perfectly matches the prior, why is the posterior probability changed? where is wrong with this formulation?

I know a better approach is to model $\theta'$ as the probability of seeing heads with a beta distribution, and use the binomial distribution as the likelihood function. Then the expectation of $\theta'$ won't change when the observation matches the prior.

But I don't quite understand what is wrong of the above model.
I'm new to statistics, any help would be awesome.

Answer 1

Since you changed the question, I will repeat my previous answer, but adjusted to fit your new version:

The essential problem, as user777 says, is that your prior seems to say you start off certain that the probability of heads is $\frac34$. In other words, your prior distribution is $\pi_0(\theta) = \delta_{3/4}(\theta)$.

Your calculation of the posterior distribution should then be $$\pi(\theta\mid \text{H seen})=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \pi_0(\theta)}{\int \left(\frac56\theta+\frac16(1-\theta)\right)\pi_0(\phi)\,d\phi}=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \delta_{3/4}(\theta)}{\int \left(\frac56\phi+\frac16(1-\phi)\right) \delta_{3/4}(\phi)\,d\phi} = \delta_{3/4}(\theta)$$ so the data does not change your prior and you are still absolutely confident that $\theta=\frac34$. In fact any observed data would not have changed your prior.

If your prior had instead been for example uniform on $[\frac12,1]$ i.e. $\pi_0(\theta) = 2$ on that interval and mean $\frac34$ then you would have had a posterior distribution $$\pi(\theta\mid \text{H seen})=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \times 2}{\int_{1/2}^1 \left(\frac56\phi+\frac16(1-\phi)\right) \times 2\,d\phi} = 2\theta+\frac12$$ when $\frac12 \le \theta \le 1$ which has a mean of $\frac{37}{48}$, marginally higher than the prior's mean as a result of the observation. You initially thought the expected probability of seeing heads was $\frac{3}{4}$ though with some uncertainty, and you did see heads, so the posterior distribution for $\theta$ adjusted in a pro-heads direction.

Answer 2

I think part of the problem is that your notation is, to put it nicely, getting in your way. The symbol $\theta$ is usually used to denote the unknown probability of a head, but here you've written $p(\theta=h)$, which I would try to parse as "the probability that $\theta=h$." But $h$ is the outcome of a trial, not the probability of a head -- more directly, the occurrence of a head in a coin toss is not a probability. So that collection of symbols doesn't really make much sense in this problem.

If your prior is $p(\theta=2/3)=1$, this is a dirac mass on $2/3$, and no amount of data will shift that prior. Can you see why?

There's a very good reason that people typically use a beta prior for this problem, and that is that the beta prior has support over all valid probabilities and only over valid probabilities. Restated, the beta prior allocates some probability to $\theta=1/3$ and $\theta=2/3$ and $\theta=2/\pi$ and so on across the uncountably many real numbers in $[0,1].$ This means that the posterior of the experiment also allocates probability over the unit interval, with some probability that $\theta$ takes on each value in $[0,1]$.

I recommend starting with a good probability textbook first, and then reading the first few chapters of Gelman's Bayesian Data Analysis.

Answer 3

The essential problem, as user777 says, is that your prior seems to say you start off certain that the probability of heads is $\frac23$. In other words, your prior distribution is $\pi_0(\theta) = \delta_{2/3}(\theta)$.

Your calculation of the posterior distribution should then be $$\pi(\theta|HHT)=\dfrac{\theta^2(1-\theta) \pi_0(\theta)}{\int \phi^2(1-\phi) \pi_0(\phi)\,d\phi}=\dfrac{\theta^2(1-\theta) \delta_{2/3}(\theta)}{\int \phi^2(1-\phi) \delta_{2/3}(\phi)\,d\phi} = \delta_{2/3}(\theta)$$ so the data does not change your prior and you are still absolutely confident that $\theta=\frac23$. In fact any observed data would not have changed your prior.

If your prior had instead been for example a $\operatorname{Beta}(4, 2)$ distribution with $\pi_0(\theta) = 20 \,\theta^{3}(1-\theta)^{1}$ and mean $\frac23$ then you would have had a posterior distribution $$\pi(\theta|HHT)=\dfrac{\theta^2(1-\theta) \,20 \,\theta^{3}(1-\theta)^{1}}{\int_0^1 \phi^2(1-\phi) \, 20 \,\phi^{3}(1-\phi)^{1}\,d\phi} = 168 \,\theta^{5}(1-\theta)^{2}$$ which is a $\operatorname{Beta}(6, 3)$ distribution still with mean $\frac23$ but a narrower dispersion than the prior. If you had observed other data then the mean of the posterior distribution might also have changed.