Are analytically tractable posterior distributions exclusively the result of a conjugate relationship in Bayesian hierarchical models?

Question

I have been building a few of my own MCMC algorithms for hierarchical Bayesian models. If the posterior distribution of say $\alpha$ is analytically tractable, I sample $\alpha$ using an R function such as rgamma with the correct parameters. If the posterior for some parameter, say $\beta$, is analytically intractable, I use a Metropolis-Hastings ratio. Throughout the first seven algorithms that I've built, I've noticed that every time a parameter has an analytically tractable posterior distribution, it's in a conjugate relationship; every time I need to use an M-H ratio, it is not a conjugate.

Now, I know that conjugacy makes determining the posterior much easier, but are there ever times when one cannot analytically derive the posterior of a conjugate prior in some hierarchical model (i.e., when using latent indicator variables)? Additionally, are there occurrences where I can analytically derive a posterior distribution that is not in a conjugate relationship?

One further question, are there other relationships which always result in a known posterior distribution that is not conjugate? (Fake Example: We use a Binomial prior and the data follows a Poisson distribution. Then the distribution will always be a $\chi^2$ distribution.)

Answer 1

1. Conjugate priors are not necessarily tractable (Robert, 1994): take for instance a Beta distribution$$f(x|\alpha,\beta) = B(\alpha,\beta)^{-1} x^{\alpha-1}(1-x)^{\beta-1}\mathbb I_{(0,1)}(x)$$as the sampling distribution. A conjugate prior on $(\alpha,\beta)$ is $$\pi(\alpha,\beta|\mu,\sigma) \propto B(\alpha,\beta)^{-\sigma}\mu_1^\alpha\mu_2^\beta$$but given the complexity of the special function $B(\alpha,\beta)$ it is not something directly simulated.

2. Manageable posteriors can be found outside exponential families, as for instance with uniform priors associated with sampling distributions from exponential families, e.g. a Normal distribution. (Uniform priors and exponential families often provide a manageable posterior.) Another (counter-)example is the posterior associated with a mixture of $k$ distributions from exponential families and component-wise conjugate priors, for a small number of observations (the details are available in an old technical report of mine, cf. Diebolt & Robert, 1990, but this is simple combinatorics). But given that the complexity of the likelihood grows with the sample size outside exponential families, since there is no sufficient statistic of fixed dimension, it is unlikely there are many examples of that kind. (The mathematical result that prevents manageable posteriors to exist outside the exponential families is called the Pitman-Koopman-Darmois theorem.)