What is the rationale behind the exponential family of distributions?

Question

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family distributions for a long time, I still do not get any intuition.

$$f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\theta }}){\Big )}$$

Can anyone help me understand Why we need it in the first place? What are some advantages of modeling a response variable to be exponential family vs normal?

EDIT: By the exponential family, I meant the general class of distributions described here.

Answer 1

What are some advantages of modeling a response variable to be exponential family vs normal?

1. The exponential family is much broader than the normal. For example, what's the advantage of using a Poisson or a binomial instead of a normal? A normal's not much use if you have counts with a low mean. What about if your data are continuous but very right skew -- perhaps times or monetary amounts? The exponential family includes the normal, the binomial, the Poisson and the Gamma as special cases (among many others)

2. It incorporates a wide variety of variance-mean relationships.

3. It derives from trying to answer a question along the lines of "what distributions are functions of a sufficient statistic", and so has models can be estimated via ML using very simple sufficient statistics; this includes the usual models available in programs that fit generalized linear models. Indeed the sufficient statistic ($T(x)$) is explicit in the exponential-family density function.

4. It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions). For example you could fit a straight-line relationship to a model which specifies the conditional response has a gamma distribution, or an exponential relationship with a conditionally Gaussian response in a GLM framework.

For Bayesians the exponential family is quite interesting because all members of the exponential family have conjugate priors.

Answer 2

For me, the main motivation behind exponential family distributions is that they are the maximum entropy distribution families given a set of sufficient statistics and a support. In other words, they are minimum assumptive distribution.

For example, if you measure only the mean and variance of real-valued quantity, the least assumptive modelling choice is a normal distribution.

From a computation standpoint, there are other advantages:

• They are closed under "evidence combination". That is, the combination of two independent likelihoods from the same exponential family is always in the same exponential family and its natural parameters are merely the sum of the natural parameters of its components. This is convenient for Bayesian statistics.

• The gradient of the cross entropy between two exponential family distributions is the difference of their expectation parameters. This means that a loss function that is such a cross entropy is a so-called matching loss function, which is convenient for optimization.

Answer 3

Glen's list is good. I'm going to add 1 more application to complement his answer: deriving conjugate priors for Bayesian inference.

A core part of Bayesian inference is deriving posterior distributions $p(\theta|y) \propto p(y|\theta) p(\theta)$. Having a prior $p(\theta)$ that is conjugate to the likelihood $p(y|\theta)$ means that the posterior $p(y|\theta)$ and prior $p(\theta)$ will belong to the same class of probability distributions.

The useful property I'm referring to is that, for a likelihood of $n$ observations drawn from a one-parameter exponential family of the form

$p(y_1,\ldots,y_n|\theta) = \prod p(y_i|\theta) \propto g(\theta)^n \exp \big[ h(\theta) \sum t(y_i) \big]$,

we can simply write out a conjugate prior as

$p(\theta) \propto g(\theta)^\nu \big[ h(\theta) \delta \big]$

and then the posterior works out as

$p(\theta|y_1,\ldots,y_n) \propto g(\theta)^{n+\nu} \exp \big[ h(\theta) \big( \sum t(y_i) + \delta \big) \big]$

Why is this conjugacy useful? Because it simplifies both our interpretation and computation while performing Bayesian inference. It also means we can easily come up with analytical expressions for the posterior without having to do too much algebra.

Answer 4

You want your model of the data to reflect the generating process. The 'process' generating Gaussian variables has very different characteristics than that governing the exponential, and it's not always intuitive as to why. Sometimes you need to appreciate other distributional characteristics. As one example, consider that the hazard function for Gaussian is increasing while exponential is flat. As a trite practical example, suppose Im going to poke you at intervals, and the 'inter poke interval' will be chosen by Gaussian or exponential generating function. Under a Gaussian, you'd find that pokes are predictable, and feel highly likely after long intervals. Under exponential, they'd feel very unpredictable. The reason for this is due to the generating function, which is dependent on the underlying phenomenon.