# What is the rationale behind the exponential family of distributions?

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family distributions for a long time, I still do not get any intuition.

$$f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\theta }}){\Big )}$$

Can anyone help me understand Why we need it in the first place? What are some advantages of modeling a response variable to be exponential family vs normal?

EDIT: By the exponential family, I meant the general class of distributions described here.

• Please clarify the sense in which you are using "exponential family." Some respondents have implicitly assumed you mean a one (or maybe two) parameter family based on the distribution function $F(t)=1-e^{-t}$ (for $t \ge 0$) whereas others are assuming you mean a much, much more general form of distributional family. – whuber Feb 5 '18 at 22:51
• @whuber I think he means "exponential families" and not the family of exponential distributions since he mentions the Poisson and Gaussian distribution. – Neil G Feb 5 '18 at 22:53
• @Neil That isn't in the least clear to me: Gaussian and Poisson are learned in elementary courses, so maybe they are being contrasted with Exponential distributions. After all, the normal distribution is in the exponential family, too, which makes the closing remark about "exponential family vs. normal" difficult to interpret. I didn't want to close this thread, but since an answer has already appeared that seems to understand "exponential family" in the narrower sense I have described, I think it best to pause until the potential confusion can be eliminated. – whuber Feb 5 '18 at 22:55
• That's fair enough. – Neil G Feb 5 '18 at 23:06
• @whuber Updated the post. Hope it clarifies it. – denizen of the north Feb 6 '18 at 15:33

What are some advantages of modeling a response variable to be exponential family vs normal?

1. The exponential family is much broader than the normal. For example, what's the advantage of using a Poisson or a binomial instead of a normal? A normal's not much use if you have counts with a low mean. What about if your data are continuous but very right skew -- perhaps times or monetary amounts? The exponential family includes the normal, the binomial, the Poisson and the Gamma as special cases (among many others)

2. It incorporates a wide variety of variance-mean relationships.

3. It derives from trying to answer a question along the lines of "what distributions are functions of a sufficient statistic", and so has models can be estimated via ML using very simple sufficient statistics; this includes the usual models available in programs that fit generalized linear models. Indeed the sufficient statistic ($T(x)$) is explicit in the exponential-family density function.

4. It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions). For example you could fit a straight-line relationship to a model which specifies the conditional response has a gamma distribution, or an exponential fit with a Gaussian response in a GLM framework.

For Bayesians the exponential family is interesting (and in some cases useful) because all members of the exponential family have conjugate priors.

• I am a bit lost on your 3rd point. As long as I recall, all probability distributions from my undergrad probability class are functions of their sufficient statistics. It may not be the case for strange distributions like cauchy (whose sufficient statistic I am not so sure) or other power law distributions. But why is this a big deal? – denizen of the north Jul 18 '18 at 19:54
• It's possible I was not expressing myself clearly. See Koopman, B.O., (1936), "On Distributions Admitting a Sufficient Statistic", Transactions of the American Mathematical Society, 39:3, 399-409. This is where the concept of the exponential family arises; the specific sense in which the exponential family is special in relation to sufficiency is explained in the first page and the first few lines of the second page. – Glen_b Jul 19 '18 at 1:25

For me, the main motivation behind exponential family distributions is that they are the maximum entropy distribution families given a set of sufficient statistics and a support. In other words, they are minimum assumptive distribution.

For example, if you measure only the mean and variance of real-valued quantity, the least assumptive modelling choice is a normal distribution.

From a computation standpoint, there are other advantages:

• They are closed under "evidence combination". That is, the combination of two independent likelihoods from the same exponential family is always in the same exponential family and its natural parameters are merely the sum of the natural parameters of its components. This is convenient for Bayesian statistics.

• The gradient of the cross entropy between two exponential family distributions is the difference of their expectation parameters. This means that a loss function that is such a cross entropy is a so-called matching loss function, which is convenient for optimization.

Glen's list is good. I'm going to add 1 more application to complement his answer: deriving conjugate priors for Bayesian inference.

A core part of Bayesian inference is deriving posterior distributions $p(\theta|y) \propto p(y|\theta) p(\theta)$. Having a prior $p(\theta)$ that is conjugate to the likelihood $p(y|\theta)$ means that the posterior $p(y|\theta)$ and prior $p(\theta)$ will belong to the same class of probability distributions.

The useful property I'm referring to is that, for a likelihood of $n$ observations drawn from a one-parameter exponential family of the form

$p(y_1,\ldots,y_n|\theta) = \prod p(y_i|\theta) \propto g(\theta)^n \exp \big[ h(\theta) \sum t(y_i) \big]$,

we can simply write out a conjugate prior as

$p(\theta) \propto g(\theta)^\nu \big[ h(\theta) \delta \big]$

and then the posterior works out as

$p(\theta|y_1,\ldots,y_n) \propto g(\theta)^{n+\nu} \exp \big[ h(\theta) \big( \sum t(y_i) + \delta \big) \big]$

Why is this conjugacy useful? Because it simplifies both our interpretation and computation while performing Bayesian inference. It also means we can easily come up with analytical expressions for the posterior without having to do too much algebra.

You want your model of the data to reflect the generating process. The 'process' generating Gaussian variables has very different characteristics than that governing the exponential, and it's not always intuitive as to why. Sometimes you need to appreciate other distributional characteristics. As one example, consider that the hazard function for Gaussian is increasing while exponential is flat. As a trite practical example, suppose Im going to poke you at intervals, and the 'inter poke interval' will be chosen by Gaussian or exponential generating function. Under a Gaussian, you'd find that pokes are predictable, and feel highly likely after long intervals. Under exponential, they'd feel very unpredictable. The reason for this is due to the generating function, which is dependent on the underlying phenomenon.

• The question was ambiguous regarding whether the OP was asking about the exponential distribution or the exponential family. Here, you are interpreting the Q as the former, whereas @Glen_b was interpreting it as the latter. The OP has now clarified their Q as about the exponential family. In light of that, would you consider editing this to be about that, or possibly deleting it? – gung Feb 6 '18 at 15:36