I have been racking my brains trying to understand Bayes theorem. So, the way I have understood is that the likelihood is the probability of observing the particular outcome given a set of parameter values. The prior describes our belief probabilities over the different parameter values. The denominator acts like a normaliser ensuring that the posterior is a valid distribution.

I have a couple of things I am not clear about:

1: The likelihood is a probability and the prior is a probability as well. So, why is the numerator already not normalised between 0 and 1? Assume I have not ignored any multiplicative constants when computing the likelihood.

2: One way I think of this denominator now is that if we have a bunch of numbers and we would like to normalize them between 0 and 1, one thing we can do is divide each of these numbers by their sum. Is this what the denominator in the Bayes theorem is affectively doing?


From a technical point of view, here is the argument:

For densities (but the argument is analogous in the discrete case), we write $$ \pi \left( \theta |y\right) =\frac{f\left( y|\theta \right) \pi \left(\theta \right) }{f(y)} $$ The norming constant can be obtained as, by writing a marginal density as a joint density and then writing the joint as conditional times marginal, with the other parameter integrated out, \begin{align*} f(y)&=\int f\left( y,\theta \right) d\theta\\ &=\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta \end{align*} It ensures integration to 1 because \begin{align*} \int \pi \left( \theta |y\right) d\theta&=\int\frac{f\left( y|\theta \right) \pi \left(\theta \right) }{\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta}d\theta\\ &=\frac{\int f\left( y|\theta \right) \pi \left(\theta \right) d\theta}{\int f\left( y|\theta \right) \pi \left(\theta \right)d\theta}\\ &=1, \end{align*} where we can "take out" the integral in the denominator because $\theta$ had already been integrated out there.

  • $\begingroup$ Thanks. Now, I see how the denominator acts as a normaliser. $\endgroup$
    – user42140
    Jun 15 '15 at 5:58
  • 2
    $\begingroup$ Also it is important to stress that likelihood is not a probability in case of continuous variables. $\endgroup$
    – Tomas
    Jun 15 '15 at 8:08
  • $\begingroup$ It's always confusing when $\theta$ isn't $\theta$; consider renaming one of the dummy variables. $\endgroup$
    – user41979
    Jun 15 '15 at 8:41
  • $\begingroup$ @Hurkyl: Which bits are you referring to? $\endgroup$
    – user42140
    Jun 15 '15 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.