11
$\begingroup$

I've identified multiple places in textbooks where the GLM is described with 5 distributions (viz., Gamma, Gaussian, Binomial, Inverse Gaussian, & Poisson). This is also exemplified in the family function in R.

Occasionally I come across references to the GLM where additional distributions are included (example). Can someone explain why these 5 are special or are always in the GLM but sometimes others are?

From what I've learned so far, the GLM distributions in the exponential family all fit into the form: $$f(y;\theta,\phi)=\exp\left\{\frac{y\theta-b(\theta)}{\phi}+c(y,\phi)\right\}$$ where $\phi$ is the dispersion parameter and $\theta$ is the canonical parameter.

Can't any distribution be transformed to fit in the GLM?

$\endgroup$
  • 5
    $\begingroup$ Clearly, uniform distribution doesn't belong to exponential family. $\endgroup$ – Zhanxiong Jul 19 '15 at 1:49
  • $\begingroup$ Nice question. E.g. what about the lognormal? $\endgroup$ – Michael M Jul 21 '15 at 16:02
  • 1
    $\begingroup$ @Zhanxiong, isn't uniform a special case of beta distribution, and beta distribution is in the exponential family? $\endgroup$ – shf8888 Jul 21 '15 at 22:03
  • $\begingroup$ @shf8888 AFAIK it's only a exponential-family distribution in the limit, when it converges to the gamma distribution. $\endgroup$ – shadowtalker Jul 21 '15 at 23:14
  • $\begingroup$ @Zhanxiong, thanks for clarifying! Apologies, you're right, with unknown bounds is not an exponential family distribution. $\endgroup$ – shf8888 Jul 23 '15 at 14:08
3
$\begingroup$

As you indicate, the qualification for using a distribution in a GLM is that it be of the exponential family (note: this is not the same thing as the exponential distribution! Although the exponential distribution, as a gamma distribution, is itself part of the exponential family). The five distributions you list are all of this family, and more importantly, are VERY common distributions, so they are used as examples and explanations.

As Zhanxiong notes, the uniform distribution (with unknown bounds) is a classic example of a non-exponential family distribution. shf8888 is confusing the general uniform distribution, on any interval, with a Uniform(0, 1). The Uniform(0,1) distribution is a special case of the beta distribution, which is an exponential family. Other non-exponential family distributions are mixture models and the t distribution.

You have the definition of the exponential family correct, and the canonical parameter is very important for using GLM. Still, I've always found it somewhat easier to understand the exponential family by writing it as:

$$f(x; \theta) = a(\theta)g(x)\exp\left[b(\theta)R(x)\right]$$

There is a more general way to write this, with a vector $\boldsymbol{\theta}$ instead of a scalar $\theta$; but the one-dimensional case explains a lot. Specifically, you must be able to factor your density's non-exponentiated part into two functions, one of unknown parameter $\theta$ but not observed data $x$ and one of $x$ and not $\theta$; and the same for the exponentiated part. It may be hard to see how, e.g., the binomial distribution can be written this way; but with some algebraic juggling, it becomes clear eventually.

We use the exponential family because it makes a lot of things much easier: for instance, finding sufficient statistics and testing hypotheses. In GLM, the canonical parameter is often used for finding a link function. Finally, a related illustration of why statisticians prefer to use the exponential family in just about every case is trying to do any classical statistical inference on, say, a Uniform($\theta_1$, $\theta_2$) distribution where both $\theta_1$ and $\theta_2$ are unknown. It's not impossible, but it's much more complicated and involved than doing the same for exponential family distributions.

$\endgroup$
  • $\begingroup$ Beta distribution with both parameters unknown is still an exponential family (but a 2-parameter exponential family). What makes you think it isn't? www2.stat.duke.edu/courses/Spring11/sta114/lec/… or wikipedia $\endgroup$ – DavidR Apr 5 '17 at 13:57
  • $\begingroup$ Thanks for pointing this out, I've changed my comment...you're right! I really don't know what I meant $\endgroup$ – Henry Apr 6 '17 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.