2
$\begingroup$

I have some trouble understanding the normality assumptions of the linear model. I have found a wealth of information already, but some of it is contradictory and I couldn't find a definite answer to my questions, unfortunately.

1) I found that the residuals need to be normally distributed in order for the OLS to yield optimal results. Does this apply to the residuals in the population (according to Andy Field in his book "Discovering Statistics using SPSS", for example) or to the sampling distribution of the residuals?

2) Given a sufficiently large sample, am I right that we could only assume approximate normality in the sampling distribution of the residuals (CLT) and not the population? That is what confuses me about question 1): If it were the residuals in the population that were relevant, the CLT wouldn't provide an answer (I think).

3) Again in Andy Field's book, it is said that the parameter estimates must have a normal sampling distribution in order to allow significance testing and calculating CIs. In many other sources I looked at, the normality of the residuals is mentioned. Which one is right? I think that the statements might be equivalent concerning deterministic predictors, but couldn't the distribution of the parameter estimates be different than the distribution of the residuals if the predictors were stochastic?

$\endgroup$
3
  • $\begingroup$ all these topics were covered multiple times on this forum. also, normality is about errors, not residuals. $\endgroup$
    – Aksakal
    Apr 18, 2018 at 14:44
  • $\begingroup$ I think there would be a little improvement if we could agree not to talk about assumptions but about ideal conditions. $\endgroup$
    – Nick Cox
    Apr 18, 2018 at 17:18
  • $\begingroup$ The book you mention twice is often discussed here on CV with some despair. For example, stats.stackexchange.com/questions/157217/… It's not authoritative. It may sell well and have bizarre humour and anecdotes, but those are uncorrelated. $\endgroup$
    – Nick Cox
    Apr 18, 2018 at 17:21

2 Answers 2

2
$\begingroup$
  1. The residuals of the sample are fixed; they have no random properties. When we refer to "population residuals" what we actually mean is residuals found from doing the experiment again a large number of times. It is these population residuals that are normally distributed. Introductory texts are usually misleading in how they describe "populations".

  2. The central limit theorem (CLT) does not apply to the residuals themselves. The mean of the residuals is 0 by definition. If the population residuals are, say, exponentially distributed, then as $n \rightarrow \infty$, you will just find more and more exponentially distributed residuals that have a mean of 0. It is the regression parameter estimates that have an approximating normal distribution in large samples when regularity conditions are met. That condition, the Lindeberg condition, I paraphrase as: you can't keep sampling outliers.

  3. If the population residuals are normally distributed, then the parameter estimates are exactly normally distributed: tests are exact for every $n$. This is a consequence of the regression parameter being an unbiased linear estimator. An unbiased linear estimator means it can be expressed as a linear combination of the $Y$s or the residuals. It's also true that any weighted combination of normal random variables is also normal. If you put those pieces together, you see that an exact normal distribution is obtained at any non-singular sample size. If the residuals aren't normal, you need a modest $n$ to ensure approximate normality of the regression parameter. Andy is right that exact or approximate normality is needed to calculate CIs and tests in the usual way.

$\endgroup$
8
  • $\begingroup$ In (3) you seem to have systematically switched the meanings of "error" and "residual." $\endgroup$
    – whuber
    Apr 18, 2018 at 15:54
  • $\begingroup$ @whuber I relaxed the term "population residuals" to just "residuals" from the definition in 1). I am focusing on a statistical rather than probabilistic interpretation of a regression model. Do you see a contradiction there? Expressing the $Y$ as a fixed part and a random part is rigorous since the OLS model consistently estimates $E[Y|X]$. (I guess I reintroduce the idea of "errors" halfway through) $\endgroup$
    – AdamO
    Apr 18, 2018 at 15:56
  • $\begingroup$ The apparent switch in meanings looks like it has considerable potential to lead to confusion rather than clarity. It might help to explain at the outset what you mean by each term and to use it consistently. $\endgroup$
    – whuber
    Apr 18, 2018 at 16:05
  • $\begingroup$ @whuber good suggestion. I suppose my line of reasoning is correct, if the terminology is unclear. I think the edits are more consistent. $\endgroup$
    – AdamO
    Apr 18, 2018 at 16:38
  • $\begingroup$ Thank you. This raises a very interesting point: your term "population residuals" could mean two distinctly different things. Let's imagine a finite bivariate population $(x_i,y_i)$ and a model of that population in which $y_i=f(x_i)+\epsilon_i.$ The $\epsilon_i$ are the errors. However, we could also apply some regression procedure, such as OLS, to the entire population, thereby producing a "population fit" of the form $y_i = \hat f(x_i) + \delta_i.$ It is reasonable to call the $\delta_i$ the "population residuals." But which of these concepts are you actually referring to? $\endgroup$
    – whuber
    Apr 18, 2018 at 17:58
0
$\begingroup$

This whole normality business come up mainly in relation to the variance of parameter estimates $\hat \beta$ in a model such as this: $$y_i=\beta_0+x_i\beta_x+\varepsilon_i$$ or in vector form $$y=X\beta+\varepsilon$$ Note, that here $\varepsilon_i$ is the errors, not the residuals $\hat\varepsilon_i$. The latter is the estimate of errors. Normality assumptions are stated in terms of errors, not residuals.

Once you got your estimates of parameters $\hat\beta$ conditioned on the sample $(y,X)$, we realize that these parameter estimates are themselves random values. So, they must come from some probability distribution. This is where normality of errors comes handy. It turns out that when the errors are normal, it's easy to derive this distribution of parameter estimates. We can study its properties, and establish that they're optimal in some sense etc.

The problem is that in practice it is often difficult to claim that the errors are normal. So, it is helpful to relax this assumption, and instead maybe just ask that the error variance is finite, for instance. In the latter case plus some other conditions such as no autocorrelation, it can be shown that the variance of parameter estimates is: $$\widehat{\operatorname{s.\!e.}}(\hat{\beta}_j) = \sqrt{s^2 (X ^T X)^{-1}_{jj}}$$ where $s^2$ is the sample variance estimator of residuals, it's an estimator of error variance. We do not need normality for this result.

Now, it turns out that under reasonable conditions, in large samples, the distribution of the parameter estimates is normal. Note, that this is different from how you interpreted the application of CLT. Here, the model errors $\varepsilon_i$ don't suddenly become normal when you increase the sample size. What happens here is that the variance of the parameter estimate $\hat\beta$, because it is a linear combination of data, is the function of the sum of variances of residuals.

Summarizing, yes, normality of errors leads to normality of parameter estimates, and otherwise makes our lives easier. However, asymptotically, the parameter estimates become normal even when errors are not normal under a set of reasonable conditions, such as Gauss Markov theorem assumtptions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.