27
$\begingroup$

Some distributions have conjugate priors and some do not. Is this distinction just an accident? That is, you do the math, and it works out one way or the other, but it does not really tell you anything important about the distribution except for the fact itself?

Or does the presence or absence of a conjugate prior reflect some deeper property of a distribution? Do distributions with conjugate priors share some other interesting property or properties that other distributions lack that causes those distributions, and not the others, to have a conjugate prior?

$\endgroup$
5
  • 3
    $\begingroup$ Well you should know that any distribution that can be written as a member of the regular exponential family must have a conjugate prior. $\endgroup$
    – user25658
    May 18, 2013 at 16:42
  • $\begingroup$ Do we know of any interesting class of distributions that have been definitely shown not to have conjugate priors? I know of very few distributions with 3 or more parameters that have known CPs, but I am not sure if we know these do not exist, or just know that we have not found them. $\endgroup$
    – andrewH
    May 21, 2013 at 3:28
  • 1
    $\begingroup$ Interesting. It could be seen as a property of the operator transporting the prior to the posterior, in the same parametric family. More interestingly perhaps, it could be seen as a closure property of the triplet (prior distribution, sampling distribution, Bayes update operator). $\endgroup$
    – JohnRos
    Jul 8, 2013 at 14:41
  • $\begingroup$ @JohnRos. I like the way you think. $\endgroup$
    – andrewH
    Aug 9, 2013 at 19:31
  • 1
    $\begingroup$ Concerning your opening statement, just be careful with the trivial case of priors that put all mass in a single value of the parameter space (not really useful for doing inference, huh?). Bayes's Theorem shows that these are conjugate priors for every model. Of course, they represent the prior knowledge of someone with "fixed ideas". $\endgroup$
    – Zen
    Aug 18, 2013 at 22:31

3 Answers 3

13
$\begingroup$

It is not by accident. Here you shall find a brief a very nice review on conjugate priors. Concretely, it mentions that if there exist a set of sufficient statistics of fixed dimension for the given likelihood function, then you can construct a conjugate prior to it. Having a set of sufficient statistics means that you can factorize the likelihood in a form that lets you estimate the parameters in a computationally efficient way.

Besides that, having conjugate priors is not only computationally convenient. It also provides smoothing and allows to work with very little samples or no previous samples, which is necessary for problems like decision making, in cases where you have very little evidence.

$\endgroup$
1
  • $\begingroup$ @LernerZhang Your edit introduced the word “soothing”—check the edit history. $\endgroup$ May 29, 2021 at 5:01
2
$\begingroup$

I am very new to Bayesian statistics, but it seems to me that all of these distributions (and if not all of them then at least those that are useful) share the property that they are described by some limited metric about the observations that define them. I.e., for a normal distribution, you don't need to know every detail about every observation, just their total count and sum.

To put it another way, assuming you already know the class/family of distribution, then the distribution has strictly lower information entropy than the observations that resulted in it.

Does this seem trivial, or is it kind of what you're looking for?

$\endgroup$
1
$\begingroup$

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

 EDIT

Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family. That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Further, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior. It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior. We don't get nice closed formulas, but we only need one numerical integration to get the normalizing constant.

Now, the usual beta prior density has one further important property: The posterior expectation is a linear function: $$ \DeclareMathOperator{\E}{\mathbb{E}} \E \left\{ \E (\theta \mid X=x)\right\} = ax+b $$ (for some $a,b$). The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker. So in these sense the usual conjugate families plays a role in Bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.

There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function. Then we could get a combined likelihood function by multiplying the prior likelihood with the data likelihood. We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.

So, summarizing, the usual conjugate families in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!

$\endgroup$
4
  • 2
    $\begingroup$ This is really a comment, not an answer, @kjetil. It should be elaborated into an answer or converted to a comment. $\endgroup$ Aug 19, 2013 at 15:01
  • 4
    $\begingroup$ @gung I am reluctant to convert this reply into a comment because it seems that it can be interpreted as an answer: it asserts that the existence of a conjugate prior is of little importance apart from simplifying computations. (I believe there may be reasons to dispute the validity of that assertion, but being incorrect is not the same as not answering!) $\endgroup$
    – whuber
    Aug 19, 2013 at 15:12
  • $\begingroup$ @whuber: what reasons apart of computational simplicity do you think of? I will try to expand on the anserv... $\endgroup$ Apr 15, 2017 at 16:46
  • 2
    $\begingroup$ Because an explicit mathematical formulation of a relationship is something that can be analyzed and understood, whereas a mere computational result is just that--a result, typically offering no generalizable insight. It's like the difference between having a map of a country that you can study and learn from, compared to having a voice-only GPS device that will give driving directions. Both will get you from one point to another, but the former will tell you much more about the space you are driving through. $\endgroup$
    – whuber
    Apr 15, 2017 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.