35
$\begingroup$

My understanding is that when using a Bayesian approach to estimate parameter values:

  • The posterior distribution is the combination of the prior distribution and the likelihood distribution.
  • We simulate this by generating a sample from the posterior distribution (e.g., using a Metropolis-Hasting algorithm to generate values, and accept them if they are above a certain threshold of probability to belong to the posterior distribution).
  • Once we have generated this sample, we use it to approximate the posterior distribution, and things like its mean.

But, I feel like I must be misunderstanding something. It sounds like we have a posterior distribution and then sample from it, and then use that sample as an approximation of the posterior distribution. But if we have the posterior distribution to begin with why do we need to sample from it to approximate it?

$\endgroup$
1
  • 2
    $\begingroup$ "We simulate this by generating a sample from the posterior distribution" Actually, we do not know the posterior distribution yet, we only know it up to a multiplicative constant (by the Bayes' Theorem). $\endgroup$
    – TrungDung
    Apr 29, 2021 at 11:47

3 Answers 3

29
$\begingroup$

This question has likely been considered already on this forum.

When you state that you "have the posterior distribution", what exactly do you mean? "Having" an available$-$in the sense I can compute it everywhere$-$function of $\theta$ that I know to be proportional to the posterior density, namely$$\pi(\theta|x) \propto \pi(\theta) \times f(x|\theta)$$as for instance with the completely artificial target$$\pi(\theta|x)\propto\exp\{-\|\theta-x\|^2-\|\theta+x\|^4-\|\theta-2x\|^6-100\|\theta\|^5\},\ \ x,\theta\in\mathbb{R}^{18}\tag{1},$$does not tell me what is

  1. the posterior expectation of a function of $\theta$, e.g., $\mathbb{E}[\mathfrak{h}(\theta)|x]$, posterior mean that operates as a Bayesian estimator under standard losses;
  2. the optimal decision under an arbitrary utility function, decision that minimizes the expected posterior loss;
  3. a 90% or 95% range of uncertainty on the parameter(s), a sub-vector of the parameter(s), or a function of the parameter(s), aka HPD region$$\{h=\mathfrak{h}(\theta);\ \pi^\mathfrak{h}(h)\ge \underline{h}\}$$
  4. the most likely model to choose between setting some components of the parameter(s) to specific values versus keeping them unknown (and random).

For instance, the fact that the rhs of (1) is known does not tell how to solve $$\int_{\mathcal H} \exp\{-\|\theta-x\|^2-\|\theta+x\|^4-\|\theta-2x\|^6-100\|\theta\|^5\}\,\text d\theta=\qquad\\0.95\int_{\mathbb R^{18}} \exp\{-\|\theta-x\|^2-\|\theta+x\|^4-\|\theta-2x\|^6-100\|\theta\|^5\}\,\text d\theta$$ and optimize over all such $\mathcal H$'s.

These items are only examples of many usages of the posterior distribution. In all cases but the simplest ones, one cannot provide answers by solely staring at the posterior distribution density as an available function and one need proceed through numerical resolutions like Monte Carlo and Markov chain Monte Carlo methods.

$\endgroup$
6
  • $\begingroup$ Thank you very much for the answer Xi'an. I'm sure this answers my question, but I'm still having a bit of difficulty grasping it. Am I right that we have a probability density function corresponding to the posterior (i.e., by combining the prior and the likelihood)? Why could we not find the 95% CI directly from this, rather than from the sampled posterior distribution? $\endgroup$
    – Dave
    Oct 16, 2017 at 19:16
  • 2
    $\begingroup$ @Dave I think the key here is what you mean by "have." In general you will not have a closed form solution, so you will not "have" the function in a useful sense. $\endgroup$
    – monk
    Jan 24, 2018 at 20:06
  • $\begingroup$ @monk thanks for the reply! Do you mind elaborating on what makes a non-closed form solution? $\endgroup$
    – Dave
    Jan 25, 2018 at 21:50
  • 5
    $\begingroup$ Suppose your prior is Beta(a, b) and your likelihood is Binomial(n, p). How do you calculate the expected value of your posterior? Try working out the integral of that product with pen and paper. In general, such an integral will be something requiring a computer to get a precise value for. Alternatively, you could discover that Beta is conjugate prior to Binomial, and therefore the posterior will be Beta (with easily computable parameters). But often you will not be so lucky. Pinning down a definition of "closed form" is hard, and worth reading about on its own. $\endgroup$
    – monk
    Jan 28, 2018 at 18:25
  • $\begingroup$ If the posterior distribution is simply a normal distribution where posterior mean and variance are known, it is still necessary to sample? $\endgroup$
    – hehe
    Jan 25, 2023 at 6:22
11
$\begingroup$

Yes you might have an analytical posterior distribution. But the core of Bayesian analysis is to marginalize over the posterior distribution of parameters so that you get a better prediction result both in terms of accuracy and generalization capability. Basically, you want to obtain a predictive distribution which has the following form.

$p(x|D)=\int p(x|w) p(w|D)dw$

where $p(w|D)$ is the posterior distribution which you might have an analytical form for. But in many cases, $p(w|D)$ has a complex form that does not belong to any known distribution family nor in conjugacy with $p(x|w)$. This makes the above integrand impossible to calculate analytically. Then you have to resort to sampling approximation of the integrand which is the entire purpose of the advanced sampling technique such as markov chain monte carlo

$\endgroup$
2
0
$\begingroup$

A short answer:

Sampling is not only used to estimate a distribution function*, it is also used to perform computations with a density function and MCMC is just one of many ways of sampling. Often such computations are a way of Monte Carlo integration.

Computing a posterior average, a 95% highest density interval, marginal distributions*, etcetera, if the density function is a complex function then such values can be difficult to derive analytically and manual integration might require a lot of computations to perform.

A sampling method is an alternative to approximate the desired quantity.


*In the case of nuisance parameters, then Bayes rule doesn't give the posterior of the parameter, but of a joint distribution with the nuisance parameter. So the statement "if we already KNOW the posterior distribution?" must be nuanced and the marginal posterior distribution is not even known. In these cases the MCMC can also be used to compute the density function (which is unknown).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.