Besides usability, is there any epistemic justification (mathematical, philosophical, heuristic, etc) for using conjugate priors? Or is it mostly just that it's usually a good enough approximation and makes things a lot easier?

  • $\begingroup$ In fact in many cases you do not need to use conjugate priors while using MCMC, e.g. stats.stackexchange.com/questions/126265/… $\endgroup$ – Tim Jun 2 '15 at 9:44
  • 2
    $\begingroup$ There is nothing limiting about using conjugate priors since discrete mixtures of conjugate priors are also conjugate and thus you have a lot of flexibility in setting up a conjugate prior. $\endgroup$ – jaradniemi Jun 2 '15 at 13:29

Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".

For example, in the Beta-Bernoulli case, the conjugate prior is Beta with density $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\beta _{0}-1} $$ This can be interpreted as the information contained in a sample of size $\underline{n}=\alpha _{0}+\beta _{0}-2$ (loosely so, as $\underline{n}$ need not be integer of course) with $\alpha _{0}-1$ successes: $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right) }{\Gamma \left( \alpha _{0}\right) \Gamma \left( \beta _{0}\right) }\theta ^{\alpha _{0}-1}\left( 1-\theta \right) ^{\underline{n}-(\alpha _{0}-1)} \propto f(y|\theta),$$ where $f(y|\theta)$ is the likelihood function.

This may give you some indication about how to pick the prior parameters: in some cases, you may be able to say that, for example, you are as sure about the fairness of a coin as if you had tossed it, say, 20 times and seen 10 heads. That is, of course, a different strength of prior belief than if you are as sure about its fairness as if you had tossed it 100 times and seen 50 heads.

  • $\begingroup$ Does every conjugate prior has such justification? I'm not sure... $\endgroup$ – Tim Jul 6 '17 at 20:56
  • $\begingroup$ My reading of the Remark on p274 of Diaconis and Ylvisaker (1979) suggests that the answer is yes. $\endgroup$ – Christoph Hanck Jul 7 '17 at 7:56

By a result due to Diaconis and Ylvisaker (1979), we know that in the setting of a likelihood being an exponential family, linear estimators are Bayes if and only if the prior is conjugate.

This suggests some fundamental important of using the conjugate prior when the estimator turns out to be linear.

  • $\begingroup$ nts: I saw this result in ch 2.3 of Johnstone's book $\endgroup$ – user795305 Jul 6 '17 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.