# What is meant by the term 'exponential family'? Why it is named so?

I have come across the term exponential family.

The Bernoulli, Gaussian and many more distributions come under this exponential family.

What would be the commonalities between them?

-

They're called "exponential family" because they can all be written in the simple form

$$f_X(x|\theta) = \exp \left (\eta(\theta) \cdot T(x) - A(\theta) + B(x) \right )$$

There are some other, equivalent forms that are perhaps more often used, but I think that form makes the "exponential" part clearest.

See, for example, this section of the Wikipedia page on the Exponential family.

In particular, in the mid 1930s a number of authors discussed what conditions would be required for a distribution to have a sufficient statistic; Koopman$^{[1]}$ stated it would have to be "of the very special exponential type of formula (4) below" (emphasis mine), where equation (4) was equivalent to the above form.

So that form succinctly expresses what they all have in common. But the consequence of that form is that this particular class of distributions has some very nice properties; for example, $T$ is a sufficient statistic - it carries all of the information in the data about $\theta$.

A number of additional properties they all share are summarized here.

Commonly used members include the Gaussian, Poisson, binomial, and gamma (including exponential and chi-squared), but I've also had occasion to use other members (such as Tweedie with specified $p$, and inverse Gaussian).

The shared properties make some standardization of the treatment of them possible, leading to the wide use of generalized linear models (GLMs).

[1] Koopman, B.O., (1936),
"On Distributions Admitting a Sufficient Statistic",
Tranactions of the American Mathematical Society, 39:3, 399-409.

-

Just a quick tip:

http://youtu.be/nLKOQfKLUks?list=PLJ_CMbwA6bT-n1W0mgOlYwccZ-j6gBXqE

I found this lecture from Andrew Ng very useful to get the concept of exponential family distributions down,.. like sweet Kool-Aid.

-

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

If there's anything in the video that answers the question, you should succinctly summarize/paraphrase the relevant information. We try to avoid link-only answers. –  Glen_b Aug 18 '14 at 4:36