0
$\begingroup$

I'm studying Bayesian inference with the famous Bayes formula. To calculate the posterior, we need prior, likelihood and (sometimes) evidence. When I say "sometimes" it's because there are cases for which we don't need this normalization constant and we make the assumption that the posterior distribution can be approximated by the prior multiplied by the likelihood.

Why do we make this assumption when there are methods (mcmc) for calculating this integral? Is it because there are sometimes conjugate priors and so we already know approximately the posterior distribution without needing to calculate the evidence?

Thank you for your help

$\endgroup$
7
  • 2
    $\begingroup$ You always need the normalizing constant to make the posterior a valid probability density. However, the posterior is proportional to the product of prior and likelihood and there are some cases where working with the unnormalized density is fine (like in some MCMC approaches). $\endgroup$ Commented Mar 14 at 14:05
  • $\begingroup$ Thank you, so in any case, it's always better to know this normalization constant? For example, in what cases could we work with unnormalized posterior? Taking the beta-binomial distribution as an example? $\endgroup$
    – Guillaume
    Commented Mar 14 at 14:17
  • $\begingroup$ If you want to deal with the posterior directly (e.g. compute areas under its surface, or sample from it exactly) then you will need the normalizing constant. Most MCMC techniques do not require the normalizing constant, so if you want to approximately sample from the posterior using MCMC, then you probably won't need it. $\endgroup$ Commented Mar 14 at 14:44
  • $\begingroup$ Related question. $\endgroup$
    – Durden
    Commented Mar 14 at 17:24
  • $\begingroup$ E.g., you would need the evidence when doing model comparison (Bayes factors), see e.g. stats.stackexchange.com/questions/497745/… $\endgroup$ Commented Mar 14 at 17:27

1 Answer 1

4
$\begingroup$

the posterior distribution can be approximated by the prior multiplied by the likelihood

This is incorrect. The posterior density $\pi(\theta|x)$ is exactly determined by the product $\pi(\theta)\ell(\theta|x)$ in the sense that $$\pi(\theta|x)=\dfrac{\pi(\theta)\ell(\theta|x)}{\underbrace{\int\pi(\theta)\ell(\theta|x)\text d\theta}_\text{evidence}}$$

there are methods (mcmc) for calculating this integral

This is also incorrect in that (a) MCMC simulations from the posterior [which indeed do not require the evidence to be known] are rarely helpful to compute the evidence, witness the poor performances of the harmonic mean estimator and (b) simulations only produce an approximation to the integral.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.