Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I represent a biased coin with a discrete distribution $p(\theta)$, where $p(\theta=h)=\pi$ is the probability of heads, and $p(\theta=t)=1-\pi$ the probability of tails in one toss.

I have a terrible eye sight, so I have $1/6$ probability of getting the wrong observation.

Suppose I have a prior $p(\theta=h)=3/4$. Then the expectation of observing heads in one toss is then $E(x)=\frac{3}{4}\times\frac{5}{6}+\frac{1}{4}\times\frac{1}{6}=2/3$.

And I toss this coin three times and observe $x=\{head, head, tail\}$.

I want to compute the posterior probability $p(\theta|x)\propto p(x|\theta)p(\theta)$.

So the likelihood function in this case is $p(x|\theta=h)=(5/6)^2(1/6)$, $p(x|\theta=t)=(1/6)^2(5/6)$, hence the posterior probability is
$p(\theta=h|x)=\frac{p(x|\theta=h)p(\theta=h)}{p(x)}=\frac{(5/6)^2(1/6)(2/3)}{norm}=10/11$ $p(\theta=t|x)=\frac{p(x|\theta=t)p(\theta=t)}{p(x)}=\frac{(1/6)^2(5/6)(1/3)}{norm}=1/11$.

It seems that the observation perfectly matches the prior, why is the posterior probability changed? where is wrong with this formulation?

I know a better approach is to model $\theta'$ as the probability of seeing heads with a beta distribution, and use the binomial distribution as the likelihood function. Then the expectation of $\theta'$ won't change when the observation matches the prior.

But I don't quite understand what is wrong of the above model.
I'm new to statistics, any help would be awesome.

Edit:
Also I've realized I've made a mistake about the likelihood in the original post, so I changed the question a bit.

share|improve this question
1  
You might want to add the [self-study] tag. It will really help you to define $\theta$. – jaradniemi Mar 1 at 12:45
    
@jaradniemi lol added – dontloo Mar 1 at 13:04
up vote 3 down vote accepted

Since you changed the question, I will repeat my previous answer, but adjusted to fit your new version:

The essential problem, as user777 says, is that your prior seems to say you start off certain that the probability of heads is $\frac34$. In other words, your prior distribution is $\pi_0(\theta) = \delta_{3/4}(\theta)$.

Your calculation of the posterior distribution should then be $$\pi(\theta\mid \text{H seen})=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \pi_0(\theta)}{\int \left(\frac56\theta+\frac16(1-\theta)\right)\pi_0(\phi)\,d\phi}=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \delta_{3/4}(\theta)}{\int \left(\frac56\phi+\frac16(1-\phi)\right) \delta_{3/4}(\phi)\,d\phi} = \delta_{3/4}(\theta)$$ so the data does not change your prior and you are still absolutely confident that $\theta=\frac34$. In fact any observed data would not have changed your prior.

If your prior had instead been for example uniform on $[\frac12,1]$ i.e. $\pi_0(\theta) = 2$ on that interval and mean $\frac34$ then you would have had a posterior distribution $$\pi(\theta\mid \text{H seen})=\dfrac{\left(\frac56\theta+\frac16(1-\theta)\right) \times 2}{\int_{1/2}^1 \left(\frac56\phi+\frac16(1-\phi)\right) \times 2\,d\phi} = 2\theta+\frac12$$ when $\frac12 \le \theta \le 1$ which has a mean of $\frac{37}{48}$, marginally higher than the prior's mean as a result of the observation. You initially thought the expected probability of seeing heads was $\frac{3}{4}$ though with some uncertainty, and you did see heads, so the posterior distribution for $\theta$ adjusted in a pro-heads direction.

share|improve this answer
    
Thanks for the deliberate answer! – dontloo Mar 2 at 2:55

I think part of the problem is that your notation is, to put it nicely, getting in your way. The symbol $\theta$ is usually used to denote the unknown probability of a head, but here you've written $p(\theta=h)$, which I would try to parse as "the probability that $\theta=h$." But $h$ is the outcome of a trial, not the probability of a head -- more directly, the occurrence of a head in a coin toss is not a probability. So that collection of symbols doesn't really make much sense in this problem.

If your prior is $p(\theta=2/3)=1$, this is a dirac mass on $2/3$, and no amount of data will shift that prior. Can you see why?

There's a very good reason that people typically use a beta prior for this problem, and that is that the beta prior has support over all valid probabilities and only over valid probabilities. Restated, the beta prior allocates some probability to $\theta=1/3$ and $\theta=2/3$ and $\theta=2/\pi$ and so on across the uncountably many real numbers in $[0,1].$ This means that the posterior of the experiment also allocates probability over the unit interval, with some probability that $\theta$ takes on each value in $[0,1]$.

I recommend starting with a good probability textbook first, and then reading the first few chapters of Gelman's Bayesian Data Analysis.

share|improve this answer
    
Thank you so much, and I've found a mistake in the original post and changed the question a bit (theta changed to 3/4 and added a noisy observation) to let it make more sense, would you please edit the answer a bit accordingly? thanks! – dontloo Mar 1 at 16:25
    
@dontloo Your edit does not resolve the underlying conceptual problems with the post. – General Abrial Mar 1 at 16:59
1  
Absolutely agree with "your notation is, to put it nicely, getting in your way" – Adrian Mar 2 at 11:07

The essential problem, as user777 says, is that your prior seems to say you start off certain that the probability of heads is $\frac23$. In other words, your prior distribution is $\pi_0(\theta) = \delta_{2/3}(\theta)$.

Your calculation of the posterior distribution should then be $$\pi(\theta|HHT)=\dfrac{\theta^2(1-\theta) \pi_0(\theta)}{\int \phi^2(1-\phi) \pi_0(\phi)\,d\phi}=\dfrac{\theta^2(1-\theta) \delta_{2/3}(\theta)}{\int \phi^2(1-\phi) \delta_{2/3}(\phi)\,d\phi} = \delta_{2/3}(\theta)$$ so the data does not change your prior and you are still absolutely confident that $\theta=\frac23$. In fact any observed data would not have changed your prior.

If your prior had instead been for example a $\operatorname{Beta}(4, 2)$ distribution with $\pi_0(\theta) = 20 \,\theta^{3}(1-\theta)^{1}$ and mean $\frac23$ then you would have had a posterior distribution $$\pi(\theta|HHT)=\dfrac{\theta^2(1-\theta) \,20 \,\theta^{3}(1-\theta)^{1}}{\int_0^1 \phi^2(1-\phi) \, 20 \,\phi^{3}(1-\phi)^{1}\,d\phi} = 168 \,\theta^{5}(1-\theta)^{2}$$ which is a $\operatorname{Beta}(6, 3)$ distribution still with mean $\frac23$ but a narrower dispersion than the prior. If you had observed other data then the mean of the posterior distribution might also have changed.

share|improve this answer
    
Thank you so much, and I've found a mistake in the original post and changed the question a bit (theta changed to 3/4 and added a noisy observation) to let it make more sense, would you please edit the answer a bit accordingly? thanks! – dontloo Mar 1 at 16:25
    
It is not good practice to change a question so the previous answers from @user777 and me look wrong, but I have added another answer for your new version – Henry Mar 1 at 21:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.