Justification for conjugate prior?

Besides usability, is there any epistemic justification (mathematical, philosophical, heuristic, etc) for using conjugate priors? Or is it mostly just that it's usually a good enough approximation and makes things a lot easier?

• In fact in many cases you do not need to use conjugate priors while using MCMC, e.g. stats.stackexchange.com/questions/126265/…
– Tim
Jun 2, 2015 at 9:44
• There is nothing limiting about using conjugate priors since discrete mixtures of conjugate priors are also conjugate and thus you have a lot of flexibility in setting up a conjugate prior. Jun 2, 2015 at 13:29

Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".

For example, in the Beta-Bernoulli case, the conjugate prior is Beta with density $$\pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\beta _{0}-1}$$ This can be interpreted as the information contained in a sample of size $\underline{n}=\alpha _{0}+\beta _{0}-2$ (loosely so, as $\underline{n}$ need not be integer of course) with $\alpha _{0}-1$ successes: $$\pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\underline{n}-(\alpha _{0}-1)} \propto f(y|\theta),$$ where $f(y|\theta)$ is the likelihood function.

This may give you some indication about how to pick the prior parameters: in some cases, you may be able to say that, for example, you are as sure about the fairness of a coin as if you had tossed it, say, 20 times and seen 10 heads. That is, of course, a different strength of prior belief than if you are as sure about its fairness as if you had tossed it 100 times and seen 50 heads.

• Does every conjugate prior has such justification? I'm not sure...
– Tim
Jul 6, 2017 at 20:56
• My reading of the Remark on p274 of Diaconis and Ylvisaker (1979) suggests that the answer is yes. Jul 7, 2017 at 7:56

By a result due to Diaconis and Ylvisaker (1979), we know that in the setting of a likelihood being an exponential family, linear estimators are Bayes if and only if the prior is conjugate.

This suggests some fundamental important of using the conjugate prior when the estimator turns out to be linear.

• nts: I saw this result in ch 2.3 of Johnstone's book Jul 6, 2017 at 18:36