# Why do we use GLM?

I'm trying to justify using of GLM model in my project instead of a simple linear regression. A lot of sources that I've seen contain the statement that "GLM allow us to build regression models for response variables that are not normally distributed". I can't understand what that is supposed to mean since we don't make any assumptions about variable distribution. Could you, please, explain why it's often more appropriate (theoretically) to use GLM if your response variable is, say, count.

• I think your question is directing people to give you an introduction to Poisson regression. Why don't you revise your question and really tell us what you are trying to justify (in the first sentence?) Is that count variable you're working on? – Penguin_Knight Jun 23 '14 at 13:49
• Related, if not an outright duplicate: stats.stackexchange.com/questions/3024/… – Stephan Kolassa Jun 23 '14 at 13:49
• The most popular answer to that question also gives the same answer - a variable is assumed to be normally distributed. As far as I know it's not necessary for fitting the model. And, yeah, my variable is count, but I would like to know why not-normal variables don't often work good enough with simple linear regression – Evgenii Nikitin Jun 23 '14 at 13:54
• OLS does not needs the outcome to be normal, it does, however, assume the residual to be normal, or the standard error can be biased, leading to a wrong p-value. Some OLS models using count outcome tend to have non-normal residual (esp. when counts are low or zeroes are abundant,) and hence using Poisson family to model the error is a better choice. – Penguin_Knight Jun 23 '14 at 14:02
• Is the only consequence the wrong p-value? Doesn't it affect fit accuracy? – Evgenii Nikitin Jun 23 '14 at 14:40

You do make assumptions about response variable distribution in normal linear models. Normal linear models can be written as $$y_i = x_i'\beta + \varepsilon_i,\qquad \varepsilon_i\sim\mathcal{N}(0,\sigma^2)$$ but that means that $y_i$ is a linear transformation of a normally distributed random variable, thus its distribution is $$y_i\sim\mathcal{N}(x_i'\beta,\sigma^2)$$ BTW, it is this normal distribution that makes $t$- and $F$-tests possible: $\hat\beta$ is normally distributed because it is a linear transformation of $y$, $\hat\beta=(X'X)^{-1}X'y$.

Further, that means that $y_i$ is continuous and can be positive, zero, or negative.

When you have a binary response variable, it can't be normally distributed. You are actually interested in the probability $P(y=1)$ vs $P(y=0)$. You could try a linear probability model, i.e. $$P(y=1\mid x)=p(x)=x'\beta$$ where \begin{align}E(y\mid x)&=1\cdot p(x)+0\cdot(1-p(x))=p(x)=x'\beta\\ V(y\mid x) &= p(x)(1-p(x))=x'\beta(1-x'\beta)\end{align} but

• the error is not normally distributed, since it can just take two vales: $$\varepsilon=y-x'\beta\quad\Rightarrow\quad \varepsilon=\begin{cases} 1-x'\beta & \text{if } y=1\\-x'\beta & \text{if } y=0\end{cases}$$
• the error is not homoscedastic: $$V(\varepsilon)=E(\varepsilon^2)=x'\beta(1-x'\beta)$$ thus the error variance depends on $x$
• $E(y\mid x)=p(x)$ is a probability, thus it must be greater than 0 and lesser than 1, but the linear predictor $E(y\mid x)=x'\beta$ can take any real value; you could try: $$p(x)=\begin{cases} 0 & \text{if } x'\beta<0 \\ x'\beta & \text{if } 0\le x'\beta \le 1 \\ 1 & \text{if } x'\beta > 1 \end{cases}$$ but the graph of $p(x)$ would be a broken line. How could you interpret it? What could its vertices mean? (BTW: interpretation is even more important than p-values: high p-values which make sense are better than low p-values which do not make sense.)

This is why one looks for a smooth function which maps $[0,1]$ to $\mathbb{R}$: $$x'\beta=g(E(y\mid x))$$ and you get: $$P(y=1\mid x)=G(x'\beta),\qquad G(x'\beta)=g^{-1}(x'\beta)$$ In logit models, $$g(x)=\ln\left(\frac{x}{1-x}\right),\qquad g^{-1}(x)=\frac{1}{1+e^{-x}}$$ where $g^{-1}$ is the cdf of a logistic random variable.

When you have to model count data, a natural (not always perfect) solution is: $$P(y=k)=\frac{e^{-\theta}\theta^k}{k!}\quad\text{i.e.}\quad y\sim\text{Poisson}(\theta)$$ so that $$E(y\mid x)=\theta=\exp(x'\beta)$$ thus $E(y\mid x) > 0$ even if $x'\beta < 0$. Other solutions are zero-inflated or negative-binomial models.

• You are welcome. And generalized linear models are a fascinating topic. I'd recommend Gelman & Hill for an accessible introduction, then Agresti or McCullagh & Nelder for in-depth analysis. – Sergio Jun 23 '14 at 20:38