Justification for conjugate prior?

Question

Besides usability, is there any epistemic justification (mathematical, philosophical, heuristic, etc) for using conjugate priors? Or is it mostly just that it's usually a good enough approximation and makes things a lot easier?

Answer 1

Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".

For example, in the Beta-Bernoulli case, the conjugate prior is Beta with density $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\beta _{0}-1} $$ This can be interpreted as the information contained in a sample of size $\underline{n}=\alpha _{0}+\beta _{0}-2$ (loosely so, as $\underline{n}$ need not be integer of course) with $\alpha _{0}-1$ successes: $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\underline{n}-(\alpha _{0}-1)} \propto f(y|\theta),$$ where $f(y|\theta)$ is the likelihood function.

This may give you some indication about how to pick the prior parameters: in some cases, you may be able to say that, for example, you are as sure about the fairness of a coin as if you had tossed it, say, 20 times and seen 10 heads. That is, of course, a different strength of prior belief than if you are as sure about its fairness as if you had tossed it 100 times and seen 50 heads.

Answer 2

By a result due to Diaconis and Ylvisaker (1979), we know that in the setting of a likelihood being an exponential family, linear estimators are Bayes if and only if the prior is conjugate.

This suggests some fundamental important of using the conjugate prior when the estimator turns out to be linear.