3
$\begingroup$

I'm confused by the phrasing I've seen about exponential families. What does it mean to say "an" exponential family. Why not "the" exponential family?

From a pdf from Berkely: "we define an exponential family of probability distributions as those distributions..."

From Statistical Inference (George Casella, Roger L. Berger): "Let $X_1, X_2, ..., X_n$ be iid observations from a pdf $f(x|\theta)$ that belongs to an exponential family..."

I've also ready things that say, "this distribution belongs to an exponential family..."

But what are the different exponential families? Why not just say the exponential family? If there are multiple exponential families, why haven't I ever seen something like, "the binomial distribution belongs to exponential family A, while this other distribution belongs to family B..."?

I've searched around, and can't find a list of these families. How can there be an exponential family if there are not more than one of them?

$\endgroup$
12
  • 2
    $\begingroup$ In these contexts, an individual exponential family is something like a normal, Poisson, gamma, etc. I was initially a bit confused about this as well... $\endgroup$ Mar 2, 2020 at 17:56
  • 3
    $\begingroup$ "Exponential family" is like "sports car", a specific class with many specific examples, yet clear distinctions from others. $\endgroup$
    – AdamO
    Mar 2, 2020 at 18:22
  • 2
    $\begingroup$ Thanks for your responses. I think the missing piece for me was that a distribution itself could be a family. I just honestly didn't understand the idea of a "family" at all; but now I understand it's just a set of pdf's with a certain form with varying parameters. So binomial(n, p) is an example of an exponential family, and binomial(100, 0.5) is a member of that family? $\endgroup$ Mar 3, 2020 at 1:49
  • 2
    $\begingroup$ This shouldn't be closed in my opinion - it's a common point to get stuck on, and it's not exactly the topic any of the linked questions. (The accepted answer to the first linked question does answer it though.) $\endgroup$
    – N. Virgo
    Mar 14, 2021 at 10:33
  • 1
    $\begingroup$ For the sake of having a concrete answer in the comments at least: a "family of distributions" is probably best thought of as a function that takes the parameters as input and returns a distribution as output. So indeed, the binomial family is an example of an exponential family, and binomial(100, 0.5) is indeed a member of that family. The phrase "the exponential family" usually refers to the set of all exponential families. It's a bit of an unfortunate term though, because it makes the word "family" mean two different things at the same time, so I prefer to avoid it. $\endgroup$
    – N. Virgo
    Mar 14, 2021 at 10:33

1 Answer 1

2
$\begingroup$

Possibly a reason for the confusing terminology is that 'the exponential family' is a family of families $f(x|\theta)$ *.

Why the name 'exponential'? Each of the families has the property that it can be factorized, and with the factor that is a function of the sufficient statistic $T(X)$ and the parameters $\theta$ being an exponential term.

$$f(x|\theta) \propto h(x) \cdot e^{T(x)\cdot \eta(\theta)}$$

  1. So we can consider the family consisting of all families $f(x|\theta)_{h,T,\eta}$.
  2. And we can consider a single specific member family $f(x|\theta)_{h,T,\eta}$.

Because they are both families, the term 'exponential' can get stuck to both of them. It is typical to call the first the exponential family, and the second an exponential family. (and exponential becomes an adjective, like in the use 'a discrete distribution family').


* if we would consider 'the exponential family' as a family of distributions $f(x)$, then it would encompass the entire space of possible distribution functions. Every possible probability distribution occurs as a specific distribution in a family of the exponential family. For every potential distribution $f(x)$ there are some $h,T,η,θ$ that match. What is relevant to describe a member of the exponential distribution family is the parameterisation. For example, we can have distribution families that, due to their parameterisation, are not in the exponential family.

$\endgroup$
8
  • $\begingroup$ Personally I am not fan of the use of 'an exponential distribution' but it is extremely common and used by almost everyone. But in the same way I don't like to write something like 'X is normally distributed' and prefer 'X is normal distributed', so what I prefer doesn't mean that it is what is convention. $\endgroup$ Mar 26 at 16:29
  • $\begingroup$ I wonder how the uses of these two uses have historically developed. $\endgroup$ Mar 26 at 16:31
  • $\begingroup$ The term family, referring to a family of distributions, tends to be introduced only in more careful/theoretical discussions. Elsewhere "The [X] distribution" can designate either the family of all [X] distributions ("the normal distribution is parametrized by its mean & variance") or all members of that family taken individually ("the normal distribution is symmetric"). (Cf "the field vole is common in England" & "the field vole lives in shallow burrows".) ... $\endgroup$ Mar 27 at 10:01
  • $\begingroup$ .... So (here goes my theory), for many, "exponential families" or "an exponential family" will be their first encounter with family. Naturally enough, they misconstrue the adjective as pertaining to a collection of families of distributions rather than to each family of distributions in the collection, & come up with "the exponential family" (perhaps helped along by the required distinction from "the exponential distribution") . Needless to say, the misnomer propagates. $\endgroup$ Mar 27 at 10:01
  • 1
    $\begingroup$ An early occurence of the term exponential is Girshick and Savage 1951 "Bayes and Minimax Estimates for Quadratic Loss Functions", and they use the term mostly as "an exponential family", and "exponential families", using the adjective for the members, but in footnote they speak of "...they deal with several specific cases of the exponential family of distributions." and use the adjective for the family of families. So this accident already happened in one of the first uses of the term 'exponential family'.... $\endgroup$ Mar 27 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.