The posterior distribution is proportional to the likelihood times the prior distribution $p(\theta|D) \propto p(D|\theta)p(\theta)$. Computing $p(D|\theta)p(\theta)$ would give the un-normalised posterior distribution and the normalising constant gives rise to a valid probability distribution of 1. However, the normalising constant is just a scaling factor for the height of the distribution and it does not in any way affect the density of the distribution because the height of the posterior are all scaled proportionately. What is the purpose of techniques like MCMC sampling to estimate the posterior distribution ?
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Computing p(D|θ)p(θ) would give the un-normalised posterior
That sounds as if MCMC was an alternative to compute the posterior distribution. However, MCMC starts to shine whenever p(D|θ)p(θ) is to complicated and cannot be computed (with reasonable effort).
Bayes books often start with simple cases where there is a conjugate prior and the result can be computed in closed form. In many real life cases such a conjugate prior does not exist or is for other reasons not feasible. Then Monte Carlos simulation can still produce usable results. It is not primarily a technique to determine scaling factors.
-
$\begingroup$ does the density of the posterior remain the same for the un-normalised posterior, because it is just a scaling factor applied. I have read that $Z$ does not need to be computed if we want a MAP estimate of $\theta$, because the $\theta_{MAP}$ does not shift when scaled. However, wouldn't all $\theta$ values have the same density as the un normalised posterior since the height of the distribution is just scaled by some constant factor ? $\endgroup$– calveeenAug 10, 2020 at 15:28
-
$\begingroup$ I am not quite shure how much of that I understand. For the point estimate no scaling is necessary. Maybe this explains the difference well towardsdatascience.com/… $\endgroup$– BernhardAug 10, 2020 at 18:13
-
$\begingroup$ In the sense that the normalising constant only scales $P(\theta|D)$ by a factor of $Z$. Since all values are scaled, the density for the unnormalised posterior is the same as the normalised posterior ? $\endgroup$– calveeenAug 11, 2020 at 1:54
$\begingroup$
$\endgroup$
MCMC methods approximate the posterior by drawing samples from it. Using these samples, one can
- form an empirical approximation to the true posterior
- calculate certain statistics like posterior mean/variance/moments
- numerically calculate hard integrals
and so on. So, finding the posterior might be seen a byproduct of what you really want to achieve via MCMC.
Analytically, if you have the unnormalized version of a density, i.e. $p(x)=cq(x)$ where $c$ is unknown, integral calculations (moments) will include $c$ as well: $$\mathbb E[X]=\int xp(x)dx=c\int x q(x)dx$$
Even if you can easily evaluate $\int x q(x)dx$, you still need to find $\int q(x)dx$.
