1
$\begingroup$

Google searches gives no results to this question and there is the opposite question in this site, which makes me think this has an intuitive response I am missing.

In most course notes and responses I find a great deal of advantages and conveniences for the use of conjugate priors. After all, we are dealing with subjective beliefs that hardly impose a specific form of complexity into its representation (and we have a great deal of flexibility of representation with the possibility of using mixtures of conjugate distributions). Why then use non-conjugate priors?

I usually see the example of replacing a gamma prior with a lognormal prior, but no rationale is given to why the pros outweigh the cons for doing so.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Conjugate priors can make it easier to find posterior distributions, but with modern computational methods it is not necessary to use conjugate priors. The choice of an appropriate prior has to do with prior knowledge and opinion. Sometimes it is easier to translate that information into a non-conjugate prior. $\endgroup$ – BruceET May 24 at 5:18
  • $\begingroup$ For some models there might not be conjugate priors. Also, conjugate priors sometimes lead to linear updating. You might want a prior that is somehow rejected if the data strongly disagrees with the prior, that might need heavier tails. $\endgroup$ – kjetil b halvorsen May 25 at 2:26
  • $\begingroup$ @BruceET what examples are there where our internal beliefs are not able to be represented by conjugate-prior? Trying to grasp the concrete and specific shortcomings that conjugate priors could have. $\endgroup$ – Kuku May 25 at 13:13
  • $\begingroup$ Not complicated. Conjugate priors for binomial data are beta family. But for someone not familiar with shapes of beta distributions it may not seem natural to choose a beta prior, $\endgroup$ – BruceET May 25 at 17:31
  • 1
    $\begingroup$ See this paper by Diaconis & Ylvisaker. I will see if I can write an answer along those lines. $\endgroup$ – kjetil b halvorsen May 26 at 22:44
0
$\begingroup$

The biggest weakness of conjugate priors is that (in certain cases) they cannot achieve the following two properties simultaneously:

  1. Proper
  2. Vague

A conjugate prior is equivalent to adding virtual points to your dataset. This is what makes them computationally efficient. But this can also make it impossible to have a vague proper prior, because it can happen that if you choose the virtual points before seeing the data, then there will always be a dataset far away from those points, such that the virtual points will unduly influence the posterior. This happens because the posterior of an exponential family is a function of sufficient statistics, and sufficient statistics can have a breakdown point of 0, meaning it only takes a single outlier to exert arbitrary influence on the statistics.

To illustrate, suppose we want to estimate the mean parameter $m$ of a normal distribution. A conjugate prior for $m$ must be a normal distribution. It is impossible to choose a normal prior on $m$ that is both proper and vague. Consider a game where you have to pick a proper normal prior, and then I get to pick a dataset. No matter what prior you pick, I can pick a dataset (of any size) whose empirical mean is sufficiently far away from the mean of that prior, such that the posterior distribution of $m$ is unduly influenced by that prior. This happens because the sufficient statistic of $m$ is the arithmetic average, which has breakdown point 0. On the other hand, if you are allowed to use non-conjugate priors, then it is easy to choose a prior on $m$ that is both proper and vague (for example, a Cauchy distribution).

The paper "Bayesian robustness modelling using regularly varying distributions" gives the mathematical arguments behind these statements.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.