20
$\begingroup$

Some distributions have conjugate priors and some do not. Is this distinction just an accident? That is, you do the math, and it works out one way or the other, but it does not really tell you anything important about the distribution except for the fact itself?

Or does the presence or absence of a conjugate prior reflect some deeper property of a distribution? Do distributions with conjugate priors share some other interesting property or properties that other distributions lack that causes those distributions, and not the others, to have a conjugate prior?

$\endgroup$
  • 3
    $\begingroup$ Well you should know that any distribution that can be written as a member of the regular exponential family must have a conjugate prior. $\endgroup$ – user25658 May 18 '13 at 16:42
  • $\begingroup$ Do we know of any interesting class of distributions that have been definitely shown not to have conjugate priors? I know of very few distributions with 3 or more parameters that have known CPs, but I am not sure if we know these do not exist, or just know that we have not found them. $\endgroup$ – andrewH May 21 '13 at 3:28
  • 1
    $\begingroup$ Interesting. It could be seen as a property of the operator transporting the prior to the posterior, in the same parametric family. More interestingly perhaps, it could be seen as a closure property of the triplet (prior distribution, sampling distribution, Bayes update operator). $\endgroup$ – JohnRos Jul 8 '13 at 14:41
  • $\begingroup$ @JohnRos. I like the way you think. $\endgroup$ – andrewH Aug 9 '13 at 19:31
  • $\begingroup$ Concerning your opening statement, just be careful with the trivial case of priors that put all mass in a single value of the parameter space (not really useful for doing inference, huh?). Bayes's Theorem shows that these are conjugate priors for every model. Of course, they represent the prior knowledge of someone with "fixed ideas". $\endgroup$ – Zen Aug 18 '13 at 22:31
7
$\begingroup$

It is not by accident. Here you shall find a brief a very nice review on conjugate priors. Concretely, it mentions that if there exist a set of of sufficient statistics of fi xed dimension for the given likelihood function, then you can construct a conjugate prior for it. Having a set of sufficient statistics means that you can factorize the likelihood in a form that let you estimate the parameters in an computational efficient way.

Besides that, having conjugate priors it is not only computationally convenient. It also provides smoothing and allows to work with very little samples or no previous samples, which is necessary for problems like decision making, in cases where you have very little evidence.

$\endgroup$
2
$\begingroup$

I am very new to Bayesian statistics, but it seems to me that all of these distributions (and if not all of them then at least those that are useful) share the property that they are described by some limited metric about the observations that define them. I.e., for a normal distribution, you don't need to know every detail about every observation, just their total count and sum.

To put it another way, assuming you already know the class/family of distribution, then the distribution has strictly lower information entropy than the observations that resulted in it.

Does this seem trivial, or is it kind of what you're looking for?

$\endgroup$
1
$\begingroup$

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

 EDIT

Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family. That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Furteh, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior. It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior. We dont get nice closed formulas, but we only need one numerical integration to get the normalizing constant.

Now, the usual beta prior density has one further important property: The posterior expectation is a linear function: $$ \DeclareMathOperator{\E}{\mathbb{E}} \E \left\{ \E (\theta \mid X=x)\right\} = ax+b $$ (for some $a,b$). The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker. So in these sense the usual conjuage families plays a role in bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.

There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function. Then we could get a combines likelihood function by multiplying the prior likelihood with the data likelihood. We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.

So, summarizing, the usual conjugate familes in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!

$\endgroup$
  • 2
    $\begingroup$ This is really a comment, not an answer, @kjetil. It should be elaborated into an answer or converted to a comment. $\endgroup$ – gung Aug 19 '13 at 15:01
  • 4
    $\begingroup$ @gung I am reluctant to convert this reply into a comment because it seems that it can be interpreted as an answer: it asserts that the existence of a conjugate prior is of little importance apart from simplifying computations. (I believe there may be reasons to dispute the validity of that assertion, but being incorrect is not the same as not answering!) $\endgroup$ – whuber Aug 19 '13 at 15:12
  • $\begingroup$ @whuber: what reasons apart of computational simplicity do you think of? I will try to expand on the anserv... $\endgroup$ – kjetil b halvorsen Apr 15 '17 at 16:46
  • 1
    $\begingroup$ Because an explicit mathematical formulation of a relationship is something that can be analyzed and understood, whereas a mere computational result is just that--a result, typically offering no generalizable insight. It's like the difference between having a map of a country that you can study and learn from, compared to having a voice-only GPS device that will give driving directions. Both will get you from one point to another, but the former will tell you much more about the space you are driving through. $\endgroup$ – whuber Apr 15 '17 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.