Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm trying to justify using of GLM model in my project instead of a simple linear regression. A lot of sources that I've seen contain the statement that "GLM allow us to build regression models for response variables that are not normally distributed". I can't understand what that is supposed to mean since we don't make any assumptions about variable distribution. Could you, please, explain why it's often more appropriate (theoretically) to use GLM if your response variable is, say, count.

share|improve this question
1  
I think your question is directing people to give you an introduction to Poisson regression. Why don't you revise your question and really tell us what you are trying to justify (in the first sentence?) Is that count variable you're working on? –  Penguin_Knight Jun 23 at 13:49
    
Related, if not an outright duplicate: stats.stackexchange.com/questions/3024/… –  Stephan Kolassa Jun 23 at 13:49
    
The most popular answer to that question also gives the same answer - a variable is assumed to be normally distributed. As far as I know it's not necessary for fitting the model. And, yeah, my variable is count, but I would like to know why not-normal variables don't often work good enough with simple linear regression –  Evgenii Nikitin Jun 23 at 13:54
4  
OLS does not needs the outcome to be normal, it does, however, assume the residual to be normal, or the standard error can be biased, leading to a wrong p-value. Some OLS models using count outcome tend to have non-normal residual (esp. when counts are low or zeroes are abundant,) and hence using Poisson family to model the error is a better choice. –  Penguin_Knight Jun 23 at 14:02
    
Is the only consequence the wrong p-value? Doesn't it affect fit accuracy? –  Evgenii Nikitin Jun 23 at 14:40

1 Answer 1

up vote 12 down vote accepted

You do make assumptions about response variable distribution in normal linear models. Normal linear models can be written as $$y_i = x_i'\beta + \varepsilon_i,\qquad \varepsilon_i\sim\mathcal{N}(0,\sigma^2)$$ but that means that $y_i$ is a linear transformation of a normally distributed random variable, thus its distribution is $$y_i\sim\mathcal{N}(x_i'\beta,\sigma^2)$$ BTW, it is this normal distribution that makes $t$- and $F$-tests possible: $\hat\beta$ is normally distributed because it is a linear transformation of $y$, $\hat\beta=(X'X)^{-1}X'y$.

Further, that means that $y_i$ is continuous and can be positive, zero, or negative.

When you have a binary response variable, it can't be normally distributed. You are actually interested in the probability $P(y=1)$ vs $P(y=0)$. You could try a linear probability model, i.e. $$P(y=1\mid x)=p(x)=x'\beta$$ where $$\begin{align}E(y\mid x)&=1\cdot p(x)+0\cdot(1-p(x))=p(x)=x'\beta\\ V(y\mid x) &= p(x)(1-p(x))=x'\beta(1-x'\beta)\end{align}$$ but

  • the error is not normally distributed, since it can just take two vales: $$\varepsilon=y-x'\beta\quad\Rightarrow\quad \varepsilon=\begin{cases} 1-x'\beta & \text{if } y=1\\-x'\beta & \text{if } y=0\end{cases}$$
  • the error is not homoscedastic: $$V(\varepsilon)=E(\varepsilon^2)=x'\beta(1-x'\beta)$$ thus the error variance depends on $x$
  • $E(y\mid x)=p(x)$ is a probability, thus it must be greater than 0 and lesser than 1, but the linear predictor $E(y\mid x)=x'\beta$ can take any real value; you could try: $$p(x)=\begin{cases} 0 & \text{if } x'\beta<0 \\ x'\beta & \text{if } 0\le x'\beta \le 1 \\ 1 & \text{if } x'\beta > 1 \end{cases}$$ but the graph of $p(x)$ would be a broken line. How could you interpret it? What could its vertices mean? (BTW: interpretation is even more important than p-values: high p-values which make sense are better than low p-values which do not make sense.)

This is why one looks for a smooth function which maps $[0,1]$ to $\mathbb{R}$: $$x'\beta=g(E(y\mid x))$$ and you get: $$P(y=1\mid x)=G(x'\beta),\qquad G(x'\beta)=g^{-1}(x'\beta)$$ In logit models, $$g(x)=\ln\left(\frac{x}{1-x}\right),\qquad g^{-1}(x)=\frac{1}{1+e^{-x}}$$ where $g^{-1}$ is the cdf of a logistic random variable.

When you have to model count data, a natural (not always perfect) solution is: $$P(y=k)=\frac{e^{-\theta}\theta^k}{k!}\quad\text{i.e.}\quad y\sim\text{Poisson}(\theta)$$ so that $$E(y\mid x)=\theta=\exp(x'\beta)$$ thus $E(y\mid x) > 0$ even if $x'\beta < 0$. Other solutions are zero-inflated or negative-binomial models.

share|improve this answer
    
Thanks a lot, now I see why normality is required. –  Evgenii Nikitin Jun 23 at 20:09
    
You are welcome. And generalized linear models are a fascinating topic. I'd recommend Gelman & Hill for an accessible introduction, then Agresti or McCullagh & Nelder for in-depth analysis. –  Sergio Jun 23 at 20:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.