Say I have a coin and I don't know what is the probability of getting heads. So I set $\ p(H) = \theta $ where $\ \theta \sim U(0,1) $

Suppose I have flipped the coin once and observed one head so according to this formula:

$$\ (1)\ \ \ f( \theta | y) = \frac{f(y|\theta)\cdot f(\theta)}{\int_{-\infty}^{\infty} f(y | \theta ) \cdot f(\theta) \ d\theta} = \frac{likelihood \cdot prior}{normalizing \ constant} $$

the probability of heads (as shown by the teacher) will be

$$\ (2) \ \ \ f(\theta \ | \ y = 1) = \frac{\theta^1(1-\theta)^{1-y}}{\int_0^1\theta^1(1-\theta)^{1-y}d\theta} = \frac{\theta}{\int_0^1 \theta d\theta} = \frac{\theta}{1/2} = 2\theta$$

now suppose I have flipped again and observed another head, what will be the posterior probability of getting head?

If I understand correctly then $\ (y_1 | \theta) \sim Bernulli(\theta) $ and I flipped the coin more than once then $\ (y | \theta) \sim Binomial(n, \theta) $ but what exactly is "normalizing constant" and why did the teacher just dropped $\ f(\theta) $ from his calculations in $\ (2) $ ?


The prior here is uniformly distributed over $[0,1]$. Therefore the probability density function (pdf) is $f(\theta)=1$ for $\theta\in[0,1]$, and $f(\theta)=0$ for $\theta\not\in[0,1]$. So the integral over $\mathbb R$ turned into the integral over $[0,1]$ where $f(\theta)=1$.

For two coin flips the probability that we observe two heads $y_1=1,y_2=1$ under fixed value of $\theta$ is $$f(y_1=1,y_2=1\mid \theta)=\mathbb P(\text{two heads}\mid \theta)=\theta^2$$ and normalizing constant is $$ \int_{-\infty}^{\infty} f(y_1=1,y_2=1 | \theta ) \cdot f(\theta) \ d\theta = \int_0^1\theta^2\cdot 1\,d\theta = \frac13. $$ So for $\theta\in[0,1]$, pdf of posterior distribution is $$ f(\theta \mid y_1=1,y_2=1) = \frac{\theta^2}{1/3}=3\theta^2. $$

  • $\begingroup$ Thanks for your answer. Can you please explain what is normalizing constant? Also, does it mean that $\ f(\theta) $ is just an indicator so if $\ \theta \in [0,1] $ it is $\ 1 $ and else it's $\ 0 $ ? Why do we need it? It's not like $\ \theta$ could be anything else than $\ [0,1] $ ?? $\endgroup$
    – bm1125
    Jan 1 '20 at 15:32
  • $\begingroup$ The function $f(\theta|y)=\theta^2$ is not a valid density on $[0,1]$ since it does not integrates to $1$. Therefore normalizing constant $c$ is needed s.t. $f(\theta|y)=\frac{\theta^2}{c}$ integrates to 1. And this constant is exactly $\int_0^1 \theta^2 d\theta=1/3$. With this constant $f(\theta|y)=3\theta^2$ is the valid density: $\int_0^1 3\theta^2 d\theta=1$. Yes, density of uniform distribution is $\mathbb 1_{[0,1]}$. $\endgroup$
    – NCh
    Jan 1 '20 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.