Normality Assumptions of the Linear Model

Question

I have some trouble understanding the normality assumptions of the linear model. I have found a wealth of information already, but some of it is contradictory and I couldn't find a definite answer to my questions, unfortunately.

1) I found that the residuals need to be normally distributed in order for the OLS to yield optimal results. Does this apply to the residuals in the population (according to Andy Field in his book "Discovering Statistics using SPSS", for example) or to the sampling distribution of the residuals?

2) Given a sufficiently large sample, am I right that we could only assume approximate normality in the sampling distribution of the residuals (CLT) and not the population? That is what confuses me about question 1): If it were the residuals in the population that were relevant, the CLT wouldn't provide an answer (I think).

3) Again in Andy Field's book, it is said that the parameter estimates must have a normal sampling distribution in order to allow significance testing and calculating CIs. In many other sources I looked at, the normality of the residuals is mentioned. Which one is right? I think that the statements might be equivalent concerning deterministic predictors, but couldn't the distribution of the parameter estimates be different than the distribution of the residuals if the predictors were stochastic?

Answer 1

1. The residuals of the sample are fixed; they have no random properties. When we refer to "population residuals" what we actually mean is residuals found from doing the experiment again a large number of times. It is these population residuals that are normally distributed. Introductory texts are usually misleading in how they describe "populations".

2. The central limit theorem (CLT) does not apply to the residuals themselves. The mean of the residuals is 0 by definition. If the population residuals are, say, exponentially distributed, then as $n \rightarrow \infty$, you will just find more and more exponentially distributed residuals that have a mean of 0. It is the regression parameter estimates that have an approximating normal distribution in large samples when regularity conditions are met. That condition, the Lindeberg condition, I paraphrase as: you can't keep sampling outliers.

3. If the population residuals are normally distributed, then the parameter estimates are exactly normally distributed: tests are exact for every $n$. This is a consequence of the regression parameter being an unbiased linear estimator. An unbiased linear estimator means it can be expressed as a linear combination of the $Y$s or the residuals. It's also true that any weighted combination of normal random variables is also normal. If you put those pieces together, you see that an exact normal distribution is obtained at any non-singular sample size. If the residuals aren't normal, you need a modest $n$ to ensure approximate normality of the regression parameter. Andy is right that exact or approximate normality is needed to calculate CIs and tests in the usual way.

Answer 2

This whole normality business come up mainly in relation to the variance of parameter estimates $\hat \beta$ in a model such as this: $$y_i=\beta_0+x_i\beta_x+\varepsilon_i$$ or in vector form $$y=X\beta+\varepsilon$$ Note, that here $\varepsilon_i$ is the errors, not the residuals $\hat\varepsilon_i$. The latter is the estimate of errors. Normality assumptions are stated in terms of errors, not residuals.

Once you got your estimates of parameters $\hat\beta$ conditioned on the sample $(y,X)$, we realize that these parameter estimates are themselves random values. So, they must come from some probability distribution. This is where normality of errors comes handy. It turns out that when the errors are normal, it's easy to derive this distribution of parameter estimates. We can study its properties, and establish that they're optimal in some sense etc.

The problem is that in practice it is often difficult to claim that the errors are normal. So, it is helpful to relax this assumption, and instead maybe just ask that the error variance is finite, for instance. In the latter case plus some other conditions such as no autocorrelation, it can be shown that the variance of parameter estimates is: $$\widehat{\operatorname{s.\!e.}}(\hat{\beta}_j) = \sqrt{s^2 (X ^T X)^{-1}_{jj}}$$ where $s^2$ is the sample variance estimator of residuals, it's an estimator of error variance. We do not need normality for this result.

Now, it turns out that under reasonable conditions, in large samples, the distribution of the parameter estimates is normal. Note, that this is different from how you interpreted the application of CLT. Here, the model errors $\varepsilon_i$ don't suddenly become normal when you increase the sample size. What happens here is that the variance of the parameter estimate $\hat\beta$, because it is a linear combination of data, is the function of the sum of variances of residuals.

Summarizing, yes, normality of errors leads to normality of parameter estimates, and otherwise makes our lives easier. However, asymptotically, the parameter estimates become normal even when errors are not normal under a set of reasonable conditions, such as Gauss Markov theorem assumtptions.