I was reading these lecture notes and came accross the definition of GLM using exponential family of distribution. The latter seemed to have a bit of ambiguity, so I've checked that's it indeed the same in other soruces, so here is my question.

One says the the distribution on $\Bbb R$ is from exponential family if its density (PDF) or probability mass function (PMF) can be expressed as follows: $$ p(y|\theta)= b(y)\mathrm{exp}(\theta \cdot T(y) -a(\theta)). \tag{1} $$ Here comes my confusion: how can we treat PDFs and PMFs on par in such case. Of course, it is possible to do formally - PDFs and PMFs are just special cases of Radon-Nikodym derivatives of the probability distributions w.r.t. Lebesgue measure and counting measure respectively. However, it does not seem that the definition of exponential family really takes care of these things. For example, a given $p$ of the shape $(1)$ can be interpreted both as PDF and as PMF, leading to completely different probability distributions. This is not to say that $p$ can be as well interpreted as a RN derivative w.r.t. some other measure different from Lebesgue or counting, thus adding even more ambgiuity to this definition. Am I missing something?


I'm not sure what you're missing except maybe the support: An exponential family is defined with a density or mass function and a support $\Omega$.

You can find a measure-theoretic definition of exponential families in: Shao, J. (2003). Mathematical Statistics. Springer. http://books.google.ca/books?id=cyqTPotl7QcC

He writes:

A parametric family $\{P_\theta: \theta \in \Theta\}$ dominated by a $\sigma$-finite measure $\nu$ on $(\Omega, \mathcal F)$ is called an exponential family if and only if

$$\frac{dP_\theta}{d\nu} = \exp\left({[T(\omega)]}^\intercal \eta(\theta) - \xi(\theta)\right)h(\omega), \qquad \omega \in \Omega,$$

where $\exp(x) = e^x$ is the exponential function, $T$ is a random $p$-vector with a fixed positive integer $p$, $\eta$ is a function from $\Theta$ to $\mathcal R^p$, $h$ is a nonnegative Borel function on $(\Omega, \mathcal F)$, and $\xi(\theta) = \log\left(\int_\Omega > e^{{[T(\omega)]}^\intercal \eta(\theta)}h(\omega)d\nu(\omega)\right)$.

(I changed his notation to write dot products as $x^\intercal y$ rather than his $x y^\intercal$, which looks to me more like an outer product.)

I think that $\nu$ then could be your counting measure in the case of a p.m.f., or a Lebesgue measure for a p.d.f., etc.

| cite | improve this answer | |
  • $\begingroup$ Excellent, I'll read the whole chapter I guess - hopefully it does answer my question. $\endgroup$ – Ulysses Dec 22 '14 at 10:51
  • 1
    $\begingroup$ @Ulysses: Ok, please let us know if it answers your question. $\endgroup$ – Neil G Dec 22 '14 at 11:46
  • $\begingroup$ Ah, I thought that the acceptance of your answer implies that :) It answers my question, indeed. $\endgroup$ – Ulysses Dec 23 '14 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.