20
$\begingroup$

So here I am studying inference. I would like that someone could enumerate the advantages of the exponential family. By exponential family, I mean the distributions which are given as \begin{align*} f(x|\theta) = h(x)\exp\left\{\eta(\theta)T(x) - B(\theta)\right\} \end{align*}

whose support doesn't depend on the parameter $\theta$. Here are some advantages I found out:

(a) It incorporates a wide variety of distributions.

(b) It offers a natural sufficient statistics $T(x)$ according to the Neyman-Fisher theorem.

(c) It makes possible to provide a nice formula for the moment generating function of $T(x)$.

(d) It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions).

Can anyone provide any other advantage?

$\endgroup$
  • 1
    $\begingroup$ to ensure generality of the answers: are there any useful PDF which are not in the exponential family? $\endgroup$ – meduz Jun 19 '19 at 8:47
22
$\begingroup$

...why should we study it and use it?

I think your list of advantages effectively answers your own question, but let me offer some meta-mathematical commentary that might elucidate this topic. Generally speaking, mathematicians like to generalise concepts and results up to the maximal point that they can, to the limits of their usefulness. That is, when mathematicians develop a concept, and find that one or more useful theorems apply to that concept, they will generally seek to generalise the concept and results more and more, until they get to the point where further generalisation would render the results inapplicable or no longer useful. As can be seen from your list, the exponential family has a number of useful theorems attached to it, and it encompasses a wide class of distributions. This is sufficient to make it a worthy object of study, and a useful mathematical class in practice.

Can anyone provide any other advantage?

This class has various good properties in Bayesian analysis. In particular, the exponential family distributions always have conjugate priors, and the resulting posterior predictive distribution has a simple form. This makes is an extremely useful class of distributions in Bayesian statistics. Indeed, it allows you to undertake Bayesian analysis using conjugate priors at an extremely high level of generality, encompassing all the distributional families in the exponential family.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ I second the nomination of "conjugate prior" as a reason to like the exponential family. Indeed, conjugate priors and sufficient statistics play very well together, so together they would be on the top of my list of reasons to use the exponential family. $\endgroup$ – Peter Leopold Jun 7 '19 at 3:30
  • $\begingroup$ Ah! A fellow Bayesian I see! $\endgroup$ – Ben Jun 7 '19 at 4:34
  • $\begingroup$ How do you know the posterior predictive has a simple form? For example, the posterior predictive of a normal model with unknown mean and variance is noncentral, scaled student's T. Is that a simple form? $\endgroup$ – Neil G Jun 7 '19 at 21:15
  • $\begingroup$ @Neil G: With IID data from an exponential family, and a conjugate prior, the predictive distribution is a ratio of two instances of the normalising function for the prior, where the denominator arguments are updated by adding the sufficient statistic and number of observations for the new data. This is a simple and general form for the predictive distribution, which is obtained by finding the normalising factor for the congugate prior (see e.g. section 9.0.5 of these notes). $\endgroup$ – Ben Jun 7 '19 at 21:41
  • 1
    $\begingroup$ Okay, I see. I've never seen this before, thanks. $\endgroup$ – Neil G Jun 9 '19 at 1:59
4
$\begingroup$

I would say the most compelling motivation for the exponential families is that they are minimum assumptive distribution given measurements. If you have a real-valued sensor whose measurements are summarized by mean and variance, then the minimum assumption you can make about its observations is that they are normally distributed. Each exponential family is the result of a similar set of assumptions.

Jaynes avers this principle of maximum entropy:

“the maximum-entropy distribution may be asserted for the positive reason that it is uniquely determined as the one which is maximally noncommittal with regard to missing information, instead of the negative one that there was no reason to think otherwise. Thus the concept of entropy supplies the missing criterion of choice…”

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.