17
$\begingroup$

I am reading the book:

Bishop, Pattern Recognition and Machine Learning (2006)

which defines the exponential family as distributions of the form (Eq. 2.194):

$$p(\mathbf x|\boldsymbol \eta) = h(\mathbf x) g(\boldsymbol \eta) \exp \{\boldsymbol \eta^\mathrm T \mathbf u(\mathbf x)\}$$

But I see no restrictions placed on $h(\mathbf x)$ or $\mathbf u(\mathbf x)$. Doesn't this mean that any distribution can be put in this form, by appropriate choice of $h(\mathbf x)$ and $\mathbf u(\mathbf x)$ (in fact only one of them has to be chosen properly!)? So how come the exponential family does not include all probability distributions? What am I missing?

Finally, a more particular question that I am interested in is this: Is the Bernoulli distribution in the exponential family? Wikipedia claims it is, but since I am obviously confused about something here, I would like to see why.

$\endgroup$
  • 3
    $\begingroup$ for the proof that the Bernoulli distribution is in the exponential family, try using the fact that $f(x; \mu) = \exp (\log( f(x; \mu)))$ and see where that gets you $\endgroup$ – jld Jul 31 '17 at 10:45
  • 1
    $\begingroup$ Just to clarify, are you asking whether any distribution can be written in this form, or whether any family of distributions can be written in this form? You seem to have gotten answers to the latter question. $\endgroup$ – Owen Jul 31 '17 at 12:41
  • $\begingroup$ @Owen Yes, I see now that this is the crucial point. Although any distribution can be written in this form (by setting $h(\mathbf x)$ appropriately, and $g=1,\mathbf u= 0$), that does not imply that any family can be written in this form. $\endgroup$ – becko Jul 31 '17 at 12:44
  • 4
    $\begingroup$ @becko, That's exactly right. The phrasing in the text, "the exponential family", is somewhat misleading, because there's not just one exponential family; rather, each choice of $(h, g, \mathbf u)$ gives rise to a family. Many authors instead say "an exponential family", making this more clear; e.g., see the Wikipedia page: en.wikipedia.org/wiki/Exponential_family $\endgroup$ – Brent Kerby Jul 31 '17 at 14:22
  • 2
    $\begingroup$ @becko I think your argument shows that any given distribution can be one member of an exponential family, but not that any family of distributions can be an exponential family. $\endgroup$ – Matthew Drury Aug 1 '17 at 4:58
21
$\begingroup$

Well, one consequence of your definition: $$p(\mathbf x|\boldsymbol \eta) = h(\mathbf x) g(\boldsymbol \eta) \exp \{\boldsymbol \eta^\mathrm T \mathbf u(\mathbf x)\}$$ is that the support of the distribution family indexed by parameter $\eta$ do not depend on $\eta$. (The support of a probability distribution is the (closure of) the least set with probability one, or in other words, where the distribution lives.) So it is enough to give a counterexample of a distribution family with support depending on the parameter, the most easy example is the following family of uniform distributions: $ \text{U}(0, \eta), \quad \eta > 0$. (the other answer by @Chaconne gives a more sophisticated counterexample).

$\endgroup$
21
$\begingroup$

Consider the non-central Laplace distribution $$ f(x; \mu, \sigma) \propto \exp \left(-| x - \mu | / \sigma \right). $$

Unless $\mu = 0$ you won't be able to write $|x - \mu|$ as an inner product between $\mu$ and some function of $x$.

The exponential family does include the vast majority of the nice named distributions that we commonly encounter, so at first it may seem like it has everything of interest, but it by no means is exhaustive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.