What is a good explanation as to why you wouldn't have to integrate to find the posterior when you use a conjugate prior. Most examples (like for instance: http://www.youtube.com/watch?v=0XD6C_MQXXE) multiply the prior times the likelihood and end with 'look how nice the (unscaled) posterior is of the same distribution as the prior and we didn't need to integrate'. But they never calculate the marginal distribution and divide by it. That kind of makes the conjugate look like a diversion from the fact that the integration problem still isn't solved at all..so why isn't the marginal distribution needed?


You can do it more explicitly if the "by inspection" trick bothers you. Suppose that $X\mid\theta\sim\mathrm{Bin}(n,\theta)$ with prior $\mathrm{Beta}(a,b)$. Then, the posterior density is proportional to $$ \theta^{x+a-1} (1-\theta)^{n-x+b-1} \, . $$ The full posterior density is $$ \pi(\theta\mid x) = A\;\theta^{x+a-1} (1-\theta)^{n-x+b-1} I_{[0,1]}(\theta) \, , $$ in which the normalization constant $A$ must be determined by the condition that the posterior density must integrate to one. Therefore, we have $$ 1 = \int_{-\infty}^\infty \pi(\theta\mid x) \, d\theta = A \int_0^1 \theta^{x+a-1} (1-\theta)^{n-x+b-1} \, d\theta $$ $$ = A \frac{\Gamma(x+a)\Gamma(n-x+b)}{\Gamma(n+a+b)} \int_0^1 \frac{\Gamma(n+a+b)}{\Gamma(x+a)\Gamma(n-x+b)} \theta^{x+a-1} (1-\theta)^{n-x+b-1} \, d\theta \, . $$ The integral above is equal to one, because you are integrating the density of a $\mathrm{Beta}(x+a,n-x+b)$ random variable. Hence, $$ A = \frac{\Gamma(n+a+b)}{\Gamma(x+a)\Gamma(n-x+b)} \, , $$ and the full posterior is $$ \pi(\theta\mid x) = \frac{\Gamma(n+a+b)}{\Gamma(x+a)\Gamma(n-x+b)}\theta^{x+a-1} (1-\theta)^{n-x+b-1} I_{[0,1]}(\theta) \, , $$ meaning that $\theta\mid X=x\sim\mathrm{Beta}(x+a,n-x+b)$.

After doing this a couple of times, you will appreciate the shortcut provided by the "by inspection" way.

  • $\begingroup$ Thanks for the elaborate answer, this backward reasoning approach I didn't see before, unfortunately I don't entirely get it. Isn't the part between the '=' and the integral sign, on the fourth formula line a bit artificial? (isn't moving 1/B(x+a,n-x+n) behind the integral sign already like moving the constant A behind the integral sign?) $\endgroup$
    – user26993
    Jun 19 '13 at 20:17

Actually the marginal is needed, it is just that the marginal is known and we don't need to integrate to find it. Think of all the trig identities in the back of the calculus book. We could re derive them by integration and the tricks used the first time, but if we find an integral that matches one of the identities then we can just look at what the table says and have the solution without the work. The same works with the conjugate prior times the likelihood, we could find the marginal by integration, but we know that integrals following a certain pattern have a given marginal, so we just plug that in rather than doing the hard work over again.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.