21
$\begingroup$

So here I am studying inference. I would like that someone could enumerate the advantages of the exponential family. By exponential family, I mean the distributions which are given as \begin{align*} f(x|\theta) = h(x)\exp\left\{\eta(\theta)T(x) - B(\theta)\right\} \end{align*}

whose support doesn't depend on the parameter $\theta$. Here are some advantages I found out:

(a) It incorporates a wide variety of distributions.

(b) It offers a natural sufficient statistics $T(x)$ according to the Neyman-Fisher theorem.

(c) It makes possible to provide a nice formula for the moment generating function of $T(x)$.

(d) It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions).

Can anyone provide any other advantage?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ to ensure generality of the answers: are there any useful PDF which are not in the exponential family? $\endgroup$
    – meduz
    Jun 19, 2019 at 8:47

2 Answers 2

Reset to default
22
$\begingroup$

...why should we study it and use it?

I think your list of advantages effectively answers your own question, but let me offer some meta-mathematical commentary that might elucidate this topic. Generally speaking, mathematicians like to generalise concepts and results up to the maximal point that they can, to the limits of their usefulness. That is, when mathematicians develop a concept, and find that one or more useful theorems apply to that concept, they will generally seek to generalise the concept and results more and more, until they get to the point where further generalisation would render the results inapplicable or no longer useful. As can be seen from your list, the exponential family has a number of useful theorems attached to it, and it encompasses a wide class of distributions. This is sufficient to make it a worthy object of study, and a useful mathematical class in practice.

Can anyone provide any other advantage?

This class has various good properties in Bayesian analysis. In particular, the exponential family distributions always have conjugate priors, and the resulting posterior predictive distribution has a simple form. This makes is an extremely useful class of distributions in Bayesian statistics. Indeed, it allows you to undertake Bayesian analysis using conjugate priors at an extremely high level of generality, encompassing all the distributional families in the exponential family.

Improve this answer
$\endgroup$
8
  • 5
    $\begingroup$ I second the nomination of "conjugate prior" as a reason to like the exponential family. Indeed, conjugate priors and sufficient statistics play very well together, so together they would be on the top of my list of reasons to use the exponential family. $\endgroup$
    – Peter Leopold
    Jun 7, 2019 at 3:30
  • $\begingroup$ Ah! A fellow Bayesian I see! $\endgroup$
    – Ben
    Jun 7, 2019 at 4:34
  • $\begingroup$ How do you know the posterior predictive has a simple form? For example, the posterior predictive of a normal model with unknown mean and variance is noncentral, scaled student's T. Is that a simple form? $\endgroup$
    – Neil G
    Jun 7, 2019 at 21:15
  • $\begingroup$ @Neil G: With IID data from an exponential family, and a conjugate prior, the predictive distribution is a ratio of two instances of the normalising function for the prior, where the denominator arguments are updated by adding the sufficient statistic and number of observations for the new data. This is a simple and general form for the predictive distribution, which is obtained by finding the normalising factor for the congugate prior (see e.g. section 9.0.5 of these notes). $\endgroup$
    – Ben
    Jun 7, 2019 at 21:41
  • 1
    $\begingroup$ Okay, I see. I've never seen this before, thanks. $\endgroup$
    – Neil G
    Jun 9, 2019 at 1:59
4
$\begingroup$

I would say the most compelling motivation for the exponential families is that they are minimum assumptive distribution given measurements. If you have a real-valued sensor whose measurements are summarized by mean and variance, then the minimum assumption you can make about its observations is that they are normally distributed. Each exponential family is the result of a similar set of assumptions.

Jaynes avers this principle of maximum entropy:

“the maximum-entropy distribution may be asserted for the positive reason that it is uniquely determined as the one which is maximally noncommittal with regard to missing information, instead of the negative one that there was no reason to think otherwise. Thus the concept of entropy supplies the missing criterion of choice…”

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.