2
$\begingroup$

I am trying to understand what makes estimating the posterior distribution such a hard problem.

So, imagine I need to estimate the posterior distribution over a set of parameters given the data y, so a quantity $P(\theta|y)$ and $\theta$ is generally high dimensional.

The prior over $\theta$ is a multivariate Gaussian i.e. $P(\theta) \sim N(\theta; 0, \Sigma)$

The likelihood i.e. $P(y|\theta)$ can be written down as product over Gaussian likelihoods.

Now, it seems to be that the posterior distribution will also be Gaussian. Is that correct?

Secondly, going through Bishop's book, it seems that the conditional posterior distributions and the marginal distributions will be Gaussian as well (assuming that the joint distribution over the parameters and data is Gaussian) and should have a closed form solution. If that is the case, why is this problem intractable?

If I need to find the parameters of this posterior distribution, can this not be set as an optimisation problem where I estimate the mean and covariance of the posterior Gaussian? I am basically having trouble visualising why this problem is complicated?

$\endgroup$
6
  • 2
    $\begingroup$ Is the reference saying that the posterior distribution for the Normal-Normal problem is intractable or is it saying IN GENERAL the problem is intractable? In general the problem is intractable, but for certain special cases ("conjugate priors") the problem is easy, and the Normal-Normal is one of those special cases. $\endgroup$
    – jbowman
    Feb 14, 2014 at 18:47
  • $\begingroup$ Thanks for the comment. Yes, it says it is intractable in general. OK, let me ask it in a different way. So, we have a likelihood with unknown mean but known variance and we know that the conjugate prior is a multivariate normal. So, of course the aim of the inference is to know the mean and the variance of the posterior distribution. So, I am given a bunch of data points. Can I estimate the posterior distribution (i.e. its mean and variance) without resorting to methods like MCMC, Variational Bayes etc. and the result will be exact? $\endgroup$
    – Luca
    Feb 14, 2014 at 22:04
  • 1
    $\begingroup$ There are two common issues when doing Bayesian analysis in (nontrivial) problems involving multiple parameters: (i) finding the joint posterior, and (ii) finding the marginal posteriors. With a nice (jointly) conjugate prior for all parameters, you might be able to write down the joint posterior just fine, but the marginals would still require integrations that may not be so easily accomplished. The nice thing about many of the computational techniques is you usually only need to be able to do something relatively simple in place of the integrations. $\endgroup$
    – Glen_b
    Feb 14, 2014 at 22:28
  • $\begingroup$ Thanks for the reply. In the case we discuss though, as the joint is Gaussian, are the marginals not Gaussian as well? i thought if I have the joint distribution with the mean vector and covariance matrix, I could just read off the corresponding mean entry and the diagonal entry in the covariance matrix to get my marginal. $\endgroup$
    – Luca
    Feb 15, 2014 at 16:32
  • $\begingroup$ Are you asking this in reference to expectation propagation? $\endgroup$
    – SomeEE
    Feb 18, 2014 at 22:53

1 Answer 1

1
$\begingroup$

Yes the posterior will be Gaussian, because normal priors are conjugate with normal likelihoods that you probably know. I will ask you to go back to your book and verify the meaning of intractability, because though the posterior has a known form (multivariate normal), sampling from the multivariate normal distribution requires MCMC techniques, which may simply be why the say it is intractable. In any case i will want to know what intractability means in your case.

$\endgroup$
5
  • $\begingroup$ Thank you for your answer. That is the problem. Books and papers all mention intractability without going too much in detail. So, now that we know that the posterior is Gaussian, we know it will have one mode which is the mean. Can we not estimate the mean and covariance of the posterior distribution using some optimisation techniques like expectation/maximization? I wonder why we need to sample from this distribution rather than find the mean/variance by formulating it as an optimisation problem. $\endgroup$
    – Luca
    Feb 14, 2014 at 15:41
  • 1
    $\begingroup$ If you have the mean and covariance matrix, sampling from a multivariate normal shouldn't require MCMC. $\endgroup$
    – Glen_b
    Feb 14, 2014 at 22:30
  • $\begingroup$ Well the software you use probably ask for the mean and variance, but the sampling behind the scene is done conditionally, most of the time by a Gibbs sampler. Yes you do need MCMC. $\endgroup$ Feb 15, 2014 at 10:28
  • $\begingroup$ Chamberlain Foncha, if you have the covariance matrix, you just calculate a matrix square root (using the Cholesky decomposition, or if $N$ is too big for that to yield stable results, the eigendecomposition) and then use a matrix-affine transformation of a product of standard normal random variates. This will work until $N$ is so big that the eigendecomposition computation outstrips one's computing resources. $\endgroup$
    – Cyan
    Feb 22, 2014 at 4:04
  • $\begingroup$ Agreed Cyan, my approach has always been, Box–Muller transform for my univariate normals (marginals) when using a Gibbs sampler. Thanks $\endgroup$ Feb 22, 2014 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.