Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have come across the term exponential family.

The Bernoulli, Gaussian and many more distributions come under this exponential family.

What would be the commonalities between them?

share|improve this question

2 Answers 2

up vote 15 down vote accepted

They're called "exponential family" because they can all be written in the simple form

$$f_X(x|\theta) = \exp \left (\eta(\theta) \cdot T(x) - A(\theta) + B(x) \right )$$

There are some other, equivalent forms that are perhaps more often used, but I think that form makes the "exponential" part clearest.

See, for example, this section of the Wikipedia page on the Exponential family.

In particular, in the mid 1930s a number of authors discussed what conditions would be required for a distribution to have a sufficient statistic; Koopman$^{[1]}$ stated it would have to be "of the very special exponential type of formula (4) below" (emphasis mine), where equation (4) was equivalent to the above form.

So that form succinctly expresses what they all have in common. But the consequence of that form is that this particular class of distributions has some very nice properties; for example, $T$ is a sufficient statistic - it carries all of the information in the data about $\theta$.

A number of additional properties they all share are summarized here.

Commonly used members include the Gaussian, Poisson, binomial, and gamma (including exponential and chi-squared), but I've also had occasion to use other members (such as Tweedie with specified $p$, and inverse Gaussian).

The shared properties make some standardization of the treatment of them possible, leading to the wide use of generalized linear models (GLMs).


[1] Koopman, B.O., (1936),
"On Distributions Admitting a Sufficient Statistic",
Tranactions of the American Mathematical Society, 39:3, 399-409.

share|improve this answer

Just a quick tip:

http://youtu.be/nLKOQfKLUks?list=PLJ_CMbwA6bT-n1W0mgOlYwccZ-j6gBXqE

I found this lecture from Andrew Ng very useful to get the concept of exponential family distributions down,.. like sweet Kool-Aid.

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

2  
If there's anything in the video that answers the question, you should succinctly summarize/paraphrase the relevant information. We try to avoid link-only answers. –  Glen_b Aug 18 at 4:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.