# What is the rationale behind the exponential family of distributions?

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family distributions for a long time, I still do not get any intuition.

$$f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\theta }}){\Big )}$$

Can anyone help me understand Why we need it in the first place? What are some advantages of modeling a response variable to be exponential family vs normal?

EDIT: By the exponential family, I meant the general class of distributions described here.

• TL;DR? Part of the reason is mathematical convenience; many problems can be solved analytically if you assume pdf are from this family. Feb 6, 2018 at 23:15

What are some advantages of modeling a response variable to be exponential family vs normal?

1. The exponential family is much broader than the normal. For example, what's the advantage of using a Poisson or a binomial instead of a normal? A normal's not much use if you have counts with a low mean. What about if your data are continuous but very right skew -- perhaps times or monetary amounts? The exponential family includes the normal, the binomial, the Poisson and the Gamma as special cases (among many others)

2. It incorporates a wide variety of variance-mean relationships.

3. It derives from trying to answer a question along the lines of "what distributions are functions of a sufficient statistic", and so has models can be estimated via ML using very simple sufficient statistics; this includes the usual models available in programs that fit generalized linear models. Indeed the sufficient statistic ($$T(x)$$) is explicit in the exponential-family density function.

4. It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions). For example you could fit a straight-line relationship to a model which specifies the conditional response has a gamma distribution, or an exponential relationship with a conditionally Gaussian response in a GLM framework.

For Bayesians the exponential family is quite interesting because all members of the exponential family have conjugate priors.

• I am a bit lost on your 3rd point. As long as I recall, all probability distributions from my undergrad probability class are functions of their sufficient statistics. It may not be the case for strange distributions like cauchy (whose sufficient statistic I am not so sure) or other power law distributions. But why is this a big deal? Jul 18, 2018 at 19:54
• It's possible I was not expressing myself clearly. See Koopman, B.O., (1936), "On Distributions Admitting a Sufficient Statistic", Transactions of the American Mathematical Society, 39:3, 399-409. This is where the concept of the exponential family arises; the specific sense in which the exponential family is special in relation to sufficiency is explained in the first page and the first few lines of the second page. Jul 19, 2018 at 1:25
• Could you explain where does the exponential function come from? I skimmed through the paper, but couldn't see the reason for it... must be some sort of Cauchy's functional equation. It would be great if you could explain in simple terms why $\exp[\dots]$. Feb 16 at 21:59
• Loosely: start with equation (3), which is a ratio of products. This leads Koopman to take logs (eq (5).) ... and that is then used in a proof (in part relating to the Jacobian of a transformation involving f) which gives an expression relating to the form of the log of f; giving the terms in the kernel of the exponential family density. I realize this is not very enlightening but this is really beyond the sort of question to ask in a comment. Feb 16 at 22:54

For me, the main motivation behind exponential family distributions is that they are the maximum entropy distribution families given a set of sufficient statistics and a support. In other words, they are minimum assumptive distribution.

For example, if you measure only the mean and variance of real-valued quantity, the least assumptive modelling choice is a normal distribution.

From a computation standpoint, there are other advantages:

• They are closed under "evidence combination". That is, the combination of two independent likelihoods from the same exponential family is always in the same exponential family and its natural parameters are merely the sum of the natural parameters of its components. This is convenient for Bayesian statistics.

• The gradient of the cross entropy between two exponential family distributions is the difference of their expectation parameters. This means that a loss function that is such a cross entropy is a so-called matching loss function, which is convenient for optimization.

Glen's list is good. I'm going to add 1 more application to complement his answer: deriving conjugate priors for Bayesian inference.

A core part of Bayesian inference is deriving posterior distributions $p(\theta|y) \propto p(y|\theta) p(\theta)$. Having a prior $p(\theta)$ that is conjugate to the likelihood $p(y|\theta)$ means that the posterior $p(y|\theta)$ and prior $p(\theta)$ will belong to the same class of probability distributions.

The useful property I'm referring to is that, for a likelihood of $n$ observations drawn from a one-parameter exponential family of the form

$p(y_1,\ldots,y_n|\theta) = \prod p(y_i|\theta) \propto g(\theta)^n \exp \big[ h(\theta) \sum t(y_i) \big]$,

we can simply write out a conjugate prior as

$p(\theta) \propto g(\theta)^\nu \big[ h(\theta) \delta \big]$

and then the posterior works out as

$p(\theta|y_1,\ldots,y_n) \propto g(\theta)^{n+\nu} \exp \big[ h(\theta) \big( \sum t(y_i) + \delta \big) \big]$

Why is this conjugacy useful? Because it simplifies both our interpretation and computation while performing Bayesian inference. It also means we can easily come up with analytical expressions for the posterior without having to do too much algebra.

You want your model of the data to reflect the generating process. The 'process' generating Gaussian variables has very different characteristics than that governing the exponential, and it's not always intuitive as to why. Sometimes you need to appreciate other distributional characteristics. As one example, consider that the hazard function for Gaussian is increasing while exponential is flat. As a trite practical example, suppose Im going to poke you at intervals, and the 'inter poke interval' will be chosen by Gaussian or exponential generating function. Under a Gaussian, you'd find that pokes are predictable, and feel highly likely after long intervals. Under exponential, they'd feel very unpredictable. The reason for this is due to the generating function, which is dependent on the underlying phenomenon.

• The question was ambiguous regarding whether the OP was asking about the exponential distribution or the exponential family. Here, you are interpreting the Q as the former, whereas @Glen_b was interpreting it as the latter. The OP has now clarified their Q as about the exponential family. In light of that, would you consider editing this to be about that, or possibly deleting it? Feb 6, 2018 at 15:36