Why is the mixtures of conjugate priors important?

Question

I have questions about the mixture of conjugate priors. I learned and saw the mixture of conjugate priors a couple of times when I am learning bayesian. I am wondering why this theorem is such important, how are we going to apply it when we are doing Bayesian analysis.

To be more specific, one theorem from Diaconis and Ylivisaker 1985 illustrated a theorem as this:

Given a sampling model $p(y|\theta)$ from an exponential family, any prior distribution can be approximated by a finite mixture of conjugate prior distributions.

More specifically, given prior $p(\theta)=\int p(\theta|\omega)p(\omega)d\omega$, we can derive the posterior:

$p(\theta|Y)\propto\int p(Y|\theta)p(\theta|\omega)p(\omega)d\omega\propto\int \frac{p(Y|\theta)p(\theta|\omega)}{p(Y|\omega)}p(Y|\omega)p(\omega)d\omega\propto \int p(\theta|Y, \omega)p(Y|\omega)p(\omega)d\omega$

Therefore,

$p(\theta|Y)=\frac{\int p(\theta|Y, \omega)p(Y|\omega)p(\omega)d\omega}{\int p(Y|\omega)p(\omega)d\omega}$

Answer 1

Calculating posteriors with general/arbitrary priors directly may be a difficult task.

On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given mixture of priors becomes the same mixture of the corresponding posteriors.

[There are also many cases where some given prior may be quite well approximated by a finite mixture of conjugate priors -- this makes for a suitable approach in some situations, when that can give approximate posteriors that are sufficiently close to the exact one.]

Answer 2

To extend @Glen_b's answer just slightly, one implication is that we can get a closed form approximation to the posterior when a non-conjugate prior is used by first approximating the non-conjugate prior with a mixture of conjugate priors and then directly solving for the posterior of the approximation.

However, in general this method seems quite tricky to use. While it's true that you can make the mixture prior arbitrarily close to the non-conjugate prior, there will generally be some error in any finite approximation. Small errors in the prior can easily propagate to huge errors in the posterior. For example, if the prior is well approximated except on the extreme tails, but the data provides strong evidence that the parameters values are in the extreme tails, these errors on the extreme tails of the prior will lead to errors in high probability regions of the posterior.