I have a questions about the mixture of conjugate priors. I learnt and say the mixture of conjugate priors a couple of times when I am learning bayesian. I am wondering why this theorem is such important, how are we going to apply it when we are doing Bayesian analysis.

To be more specific, one theorem from Diaconis and Ylivisaker 1985 illustrated a theorem as this:

Given a sampling model $p(y|\theta)$ from an exponential family, any prior distribution can be approximated by a finite mixture of conjugate prior distributions.

More specifically, given prior $p(\theta)=\int p(\theta|\omega)p(\omega)d\omega$, we can derive the posterior:

$p(\theta|Y)\propto\int p(Y|\theta)p(\theta|\omega)p(\omega)d\omega\propto\int \frac{p(Y|\theta)p(\theta|\omega)}{p(Y|\omega)}p(Y|\omega)p(\omega)d\omega\propto \int p(\theta|Y, \omega)p(Y|\omega)p(\omega)d\omega$


$p(\theta|Y)=\frac{\int p(\theta|Y, \omega)p(Y|\omega)p(\omega)d\omega}{\int p(Y|\omega)p(\omega)d\omega}$

  • 1
    $\begingroup$ This is not an answer to your question, but it is good to remember that in many cases you do not have to use conjugate priors for sampling (check here). $\endgroup$ – Tim Dec 12 '14 at 8:47
  • 2
    $\begingroup$ The theorem you quote is not true. The version you describe is about hierarchical priors, not conjugate priors. Please rephrase your question correctly. $\endgroup$ – Xi'an Dec 12 '14 at 10:35
  • $\begingroup$ @Xi'an Thanks. This quote originates from the paper <statistics.stanford.edu/sites/default/files/EFS%20NSF%20207.pdf>. It is on the bottom of page 13. $\endgroup$ – Shijia Bian Dec 12 '14 at 14:15
  • 2
    $\begingroup$ Oh, you forgot the "approximation" and the "finite" in the statement!!! "Any prior can be approximated by a finite mixture of conjugate priors" is the right quote, with the approximation not working in terms of tail behaviour. $\endgroup$ – Xi'an Dec 12 '14 at 16:05
  • 1
    $\begingroup$ @Xi'an may I also have another question? Why should we always emphasize on "finite" mixture model? In other words, is there infinite mixture model? $\endgroup$ – Shijia Bian Dec 12 '14 at 20:02

Calculating posteriors with general/arbitrary priors directly may be a difficult task.

On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given mixture of priors becomes the same mixture of the corresponding posteriors.

[There are also many cases where some given prior may be quite well approximated by a finite mixture of conjugate priors -- this makes for a very easy to apply and practical approach in many situations, that leads to approximate posteriors that may be made quite close to the exact one.]

  • 2
    $\begingroup$ The main point in Diaconis & Ylvisaker (1985) is indeed to show that finite mixtures of conjugates are (a) conjugates and (b) offer more flexibility than the original conjugates. They also require more prior information to decide about the hyperparameters, which is why they are not used that much. But it remains untrue that any prior is a mixture of conjugate priors! $\endgroup$ – Xi'an Dec 12 '14 at 10:39

To extend @Glen_b's answer just slightly, one implication is that we can get a closed form approximation to the posterior when a non-conjugate prior is used by first approximating the non-conjugate prior with a mixture of conjugate priors and then directly solving for the posterior of the approximation.

However, in general this method seems quite tricky to use. While it's true that you can make the mixture prior arbitrarily close to the non-conjugate prior, there will generally be some error in any finite approximation. Small errors in the prior can easily propagate to huge errors in the posterior. For example, if the prior is well approximated except on the extreme tails, but the data provides strong evidence that the parameters values are in the extreme tails, these errors on the extreme tails of the prior will lead to errors in high probability regions of the posterior.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.