8
$\begingroup$

When I first learned about Generalized Linear Models I thought that the assumption that the dependent variable follows some distribution from the exponential family was made to simplify calculations. However, I now read about Vector GLMs (VGLMs). VGLMs do not require the assumption that the dependent variable follows some distribution from the exponential family but they allow for a much broader set of distributions.

So my question is: WHY do we actually need the distribution assumption in GLMs?

My thoughts so far: GLMs model the mean of the assumed exponential family and thus has only one predictor (this predictor may be vector-valued in case of a vector-valued distribution mean). The variance of the distribution depends on the mean by some function and the first two moments specify the distribution uniquely within the set of all distributions from the exponential family. Thus, it is enough to specify the link function to uniquely specify the distribution. VGLMs on the other hand allow more than one predictor, one predictor for each parameter. It is therefore possible to specify the distribution by first assuming the distribution of the dependent variable and then estimate the parameters. Consider for instance the negative binomial distribution $NB(r,\mu)$. The two parameters are and $r$ (number of trials) and the mean $\mu$ (note that in this formulation $p=\frac{\mu}{\mu+r}$). Can someone verify these thoughts or give another explanation?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Specifying the canonical link function might be enough to determine the distribution. But you can use the same link function on different distributions (e.g. you can use log-link both for poisson, gamma, gaussian, inverse-gaussian, etc.) $\endgroup$
    – Maverick Meerkat
    Sep 2, 2020 at 9:53

1 Answer 1

Reset to default
6
$\begingroup$

When I discovered GLM I also wondered why it was always based on the exponential family. I have never answered to that question clearly. But...

I call $h$ the reciprocal of the link function. $\beta$ the parameter.

When I first learned about Generalized Linear Models I thought that the assumption that the dependent variable follows some distribution from the exponential family was made to simplify calculations.

Yes. I used it with stochastic gradient descent (SGD), and the update rule of SGD (the gradient) is made especially simple in the canonical GLM case. See http://proceedings.mlr.press/v32/toulis14.pdf prop 3.1 and paragraph 3.1. Finally it all works in a way that is similar to least squares (minimize average $(Y-h(\beta X))^2$) but even simpler. The interpretation of the update rule is made quite simple. For some sample $(x,y)$ :

  • compute what you expect as a mean for $y$ (that is $h(\beta x)$)
  • compare it with real observed $y$,
  • correct you parameter $\beta$ proportionality to the difference (and $x$)

Without the exp family and canonical link, the error would be multiplied by something dependant on $x$ (and maybe $y$). It would be a sort of refinement of the basic idea : varying the intensity of the correction. It gives different weights to the samples. With least square, you have to multiply by $h'(\beta x)$. Some practical tests of mine in a case with a lot of data showed it was less good (for reasons I'm incapable of explaining).

Thus, it is enough to specify the link function to uniquely specify the distribution.

Yes again.

Also pre-existing logistic regression and Poisson regression fit into the canonical GLM framework. Probably one more (historical) explanation of using the exp family + canonical link.

Maybe, "why assume the exp family in GLM" is similar to "why assume a normal noise in linear regression". For theoretical good properties and simple calculations... But does it always matter so much in practice ? Real data rarely have normal noise in cases when linear regression still works very well.

What was fundamentally useful (for me) about GLM is the difference with transformed linear regression :

  • Transformed linear regression : $E(h^{-1}(Y))=\beta X$
  • GLM : $E(Y)=h(\beta X)$

This changes everything :

  • Transformed linear regression : the estimation of the mean of $h^{-1}(Y)$ (conditionally to any function of $X$) is unbiased.
  • GLM : the estimation of the mean of $Y$ (conditionally to any function of $X$) is unbiased.

I'm not familiar with VGLM so I can't answer about it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.