I have been building a few of my own MCMC algorithms for hierarchical Bayesian models. If the posterior distribution of say $\alpha$ is analytically tractable, I sample $\alpha$ using an R function such as rgamma with the correct parameters. If the posterior for some parameter, say $\beta$, is analytically intractable, I use a Metropolis-Hastings ratio. Throughout the first seven algorithms that I've built, I've noticed that every time a parameter has an analytically tractable posterior distribution, it's in a conjugate relationship; every time I need to use an M-H ratio, it is not a conjugate.

Now, I know that conjugacy makes determining the posterior much easier, but are there ever times when one cannot analytically derive the posterior of a conjugate prior in some hierarchical model (i.e., when using latent indicator variables)? Additionally, are there occurrences where I can analytically derive a posterior distribution that is not in a conjugate relationship?

One further question, are there other relationships which always result in a known posterior distribution that is not conjugate? (Fake Example: We use a Binomial prior and the data follows a Poisson distribution. Then the distribution will always be a $\chi^2$ distribution.)


1 Answer 1

  1. Conjugate priors are not necessarily tractable (Robert, 1994): take for instance a Beta distribution$$f(x|\alpha,\beta) = B(\alpha,\beta)^{-1} x^{\alpha-1}(1-x)^{\beta-1}\mathbb I_{(0,1)}(x)$$as the sampling distribution. A conjugate prior on $(\alpha,\beta)$ is $$\pi(\alpha,\beta|\mu,\sigma) \propto B(\alpha,\beta)^{-\sigma}\mu_1^\alpha\mu_2^\beta$$but given the complexity of the special function $B(\alpha,\beta)$ it is not something directly simulated.

  2. Manageable posteriors can be found outside exponential families, as for instance with uniform priors associated with sampling distributions from exponential families, e.g. a Normal distribution. (Uniform priors and exponential families often provide a manageable posterior.) Another (counter-)example is the posterior associated with a mixture of $k$ distributions from exponential families and component-wise conjugate priors, for a small number of observations (the details are available in an old technical report of mine, cf. Diebolt & Robert, 1990, but this is simple combinatorics). But given that the complexity of the likelihood grows with the sample size outside exponential families, since there is no sufficient statistic of fixed dimension, it is unlikely there are many examples of that kind. (The mathematical result that prevents manageable posteriors to exist outside the exponential families is called the Pitman-Koopman-Darmois theorem.)

  • $\begingroup$ Thank you for referencing some papers. I definitely understand your first point, although I have a few questions regarding your second point. My interpretation of your first sentence is that one can derive posteriors if they are exponential families. Can you analytically derive every posterior distribution given a prior and data model that are within the exponential family? Additionally, it seems that conjugacy would arise in the example of a mixture of distributions solely because the mixture is designed to guarantee conjugacy. Is this typically true? $\endgroup$
    – Ron Snow
    Commented Apr 17, 2020 at 0:21
  • 1
    $\begingroup$ No and no: my first point gives a conjugate prior that is not analytically manageable and a mixture of exponential families does not provide a manageable posterior since it has a complexity of $O(k^n)$. $\endgroup$
    – Xi'an
    Commented Apr 17, 2020 at 3:59
  • $\begingroup$ I see- thank you for the clarification! $\endgroup$
    – Ron Snow
    Commented Apr 17, 2020 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.