Non-conjugate prior

Question

Can someone explain why the integral in the posterior density may not be "analytically tractable" if the prior we choose is non-conjugate?

Answer 1

Conjugacy is nice because it means that if you can deal with the pdf in the prior, you should be able to do the same with the posterior (since they're of the same form) -- but of course sometimes you want a prior that's not conjugate.

How does tractability of integrals come up in a practical Bayesian calculation?

Imagine we wish to make some inference about a parameter $\theta$:

$p(\theta|\mathbf x) \propto p(\mathbf x|\theta)\cdot p(\theta)$

where the first term on the right is the likelihood and the second term is the prior. The issue is basically to evaluate the constant of proportionality required to get a density on the right; and then you may want to be able to do various things with it (e.g. draw it; find summary statistics - its mean, or its mode, or some quantiles; perhaps even sample from it). Anyway, being able to find that integral in some way would be useful, and perhaps the most natural and obvious thing to do is attempt to find it 'algebraically' - that is, using the usual bag of tricks for evaluating integrals.

Usually, what we really mean by intractable is 'analytically intractable', but sometimes it's used a little more loosely. In some sense, "most" integrals are intractable, for various values of 'intractable' (scroll down to the discussion of integrals).

Example

As Zen points out for even that very simple example of a binomial model, there's no guarantee you can do the integration for the posterior on the parameter algebraically.

Here's a different example (a simplified version of something I've seen come up):

Consider a Bayesian posterior for the variance, $\sigma^2$ of a normal distribution with known mean $\mu$. The conjugate prior is inverse gamma, but what if we wanted a lognormal prior?

Then we'd effectively have an integral whose integrand is of the form

$$p(\sigma^2|\mu,\mathbf y)\propto p(\mathbf y|\mu,\sigma^2)\cdot p(\sigma^2)$$

where again the first term on the right of the $\propto$ is the likelihood and the second is the prior.

That likelihood is of the form:

$$f(\sigma^2; \alpha, \beta)= \frac{\beta^\alpha}{\Gamma(\alpha)}(\sigma^2)^{-\alpha - 1}\exp\left(-\frac{\beta}{\sigma^2}\right)$$

where $\alpha$ and $\beta$ are simple functions of the data, $y$, the sample size, $n$, and $\mu$, and the prior is of the form:

$$f(\sigma^2;\theta,\tau) = \frac{1}{\sigma^2 \tau \sqrt{2 \pi}}\, e^{-\frac{(\ln \sigma^2 - \theta)^2}{2\tau^2}}$$

... and the product of those is not at all algebraically "nice" to try to deal with. For example, Wolfram Alpha can't do the integral*, and it's more likely to get something like this out in a reasonable time than I am.

* (specifically, we can drop the constants and combine terms, and put $x$ for $\sigma^2$ to supply $x^{-\alpha - 2} \exp(-\frac{\beta}{x}-\frac{(\ln x - \theta)^2}{2\tau^2})$ for the integrand -- and the indefinite integral of that is what Wolfram Alpha can't do. Maybe there's a way to get it - or something else - to do the definite integral on $(0,\infty)$, though.)

Discussion of some approaches to analytical intractability

If it weren't for the fact that people so often tend to choose analytically 'nice' priors (especially when teaching the subject, but also frequently in real problems), it would be a problem that comes up almost every time. That's not to say that choosing analytically nice priors is wrong - usually we only have a vague sense of our prior information (I rarely have a specific prior distribution in mind, though I may well have some notion about possible or likely values - I may have a broad sense of where I want most of the probability on my prior to be, or very roughly where the mean might be, for example - if I don't know what specific functional-form I want for my prior and a conjugate prior can reflect the information I want to have in my prior, that may often be a quite reasonable choice).

However in a practical sense it is still quite possible to deal with this issue in a number of ways. We can, for example, approximate the posterior to varying degrees of accuracy. Here are a few examples (by no means exhaustive): (i) by approximating that desired prior in any number of ways - perhaps by a mixture of conjugate or otherwise tractable priors - yielding a corresponding mixture for the posterior, or (ii) by suitable numerical integration (which in the univariate case can work surprisingly well), or (iii) we can simulate from this distribution without knowing that integral - perhaps via rejection sampling, or via a Metropolis-Hastings type Markov Chain-Monte Carlo algorithm, as long as we have a suitable bounding function or approximant respectively).

In the past, common approaches to this issues tended to include numerical integration (or Monte Carlo integration in higher dimensions), and Laplace approximation. In fact these are still used on many problems, but we have many other tools.

Given so much Bayesian work is done using various versions of MCMC and related sampling approaches these days, analytical tractability is much less of an issue than it might once have been, even with problems with large numbers of parameters - I've seen all three of the approaches I've mentioned just above used in that context; this means we're pretty much free to choose just the prior we want, on the basis of how well it reflects our prior knowledge, or for its ability to regularize the inference - for its suitability for our inference rather than ease of algebraic manipulation. So you see, for example, Andrew Gelman advocating the use of half-Cauchy and half-t priors on variance parameters in hierarchical models, and weakly-informative Cauchy priors in logistic regression (however, that paper is not using MCMC, but rather achieving approximate inference via E-M coupled with the usual iteratively reweighted least squares for logistic regression).