Assumption of normally distributed residuals in linear regression [duplicate]

Question

Let us consider the simple linear model $y = \beta_0 + \beta_1 X + \epsilon$, where $y$ is real number, $X$ a matrix of reals and $\epsilon$ is the random "noise". The least-square estimate of the coefficients $\beta_0$ and $\beta_1$ does not depend on the nature of the "noise" term $\epsilon$. It is reasonable to assume that the expectation of $\epsilon$ is zero. For the regression problem really computes $E(y|X)$. It is also reasonable to require that $\epsilon$ has the same variance across the range of $X$. Why do we also require $\epsilon$ to be normally distributed? What goes wrong if it is not normally distributed but has a distribution symmetric around zero?

Answer 1

The normal distribution is fully defined by its mean and variance, while other distributions are not. Therefore, when residuals are centered and with fixed variance but not gaussian, you can end up with some tricky situations.

Example 1: Skewed residuals
In this example residuals come from a lognormal distribution, and I normalized them to mean 0 and std 1. It is quite evident that this is NOT a good regression model.

Example 2: Bimodal residuals (high kurtosis)
Here I sampled the residuals from a beta distribution. Again, they are centered around 0 and with variance 1 and yet, this is a very bad model.

So, all and all, you would have to fix mean and variance, but also skew and kurtosis, which means 4 total moments. I imagine also further moments can affect the goodness of fit of a regression, so in the end, the solution is fixing just two moments and asking for normality. Of course, the more moments you fix, the closer you're getting to a normal distribution, as your moment generating funciton will be getting more and more similar to a normal's.

EDIT:
As pointed out by @peuhp, the reason is that in those examples we are not maximising likelyhood when doing OLS.

Answer 2

Why do we also require ϵ to be normally distributed?

IID normally distributed $\epsilon$ are needed to ensure that the parameters $\beta_0$ and $\beta_1$ provided by ordinary least square corresponds to the maximum likelihood estimators. If it is not the case, you don't have a guaranty of optimality of the OSL solutions.

So basically, if noise is not gaussian you are using the optimal solution of a problem to solve another problem ; which may cause some troubles.

Actually, if the noise is considered symmetric wrt 0 and iid (but not necessarly gaussian) the Gauss-Markov theorem ensures OSL is the best linear unbiased estimator. However, there is no reason to limit ourself to linear estimator.

The answer given by @Davide DN give you some illustrations about what may happen in practice. Another illustrating example is when using a uniform noise on [-1,1] (so symmetric), where you can easily build settings where the OLS provides solution corresponding to a probability 0.

Answer 3

The assumptions of OLS are vague, and they only ensure optimality within a very restrictive set of estimators, specifically those which are linear estimators of the mean of a distribution. What this basically boils down to is that the best way to construct a weighted average for estimating a mean is to choose weights that are inversely proportional to the variance. This is a very restrictive consideration that does not consider, for example, the possibility that the median may also be a valid measure of location, since it is a nonlinear estimator. The reason normality is often assumed is that if you are trying to estimate the mean of the distribution, then certain general inferential practices are justified.

First, if the distribution is normal, estimation and inference based on F-tests and t-tests are valid for any sample size. Second, if the distribution is not normal, but it is symmetric, unimodal, and has finite constant variance with a reasonably large support, then for sufficiently large samples, inferences based on t-tests and F-tests will be approximately valid due to the Central Limit Theorem.

If the mean of the distribution is not a reasonable measure of location, then OLS is not a reasonable method of estimation. For example, if the errors are either bimodal or skewed, one would not use the mean to summarize such a distribution.

If the errors are such that the mean is reasonable, it may well be that another estimator that is nonlinear could be an improvement. For example, if the errors follow a Laplace Distribution, then the MLE (sample median) is an improvement over OLS. However, in practice, it can be difficult to distinguish errors that follow a Laplace Distribution from those which follow another distribution that has similar properties. Since, in these situations, there may not be a large price to incorrectly assuming normality if the sample size is sufficiently large, then the OLS estimation with normality assumptions will likely produce valid conclusions anyway.

So, to summarize, the main reasoning for normality assumptions in OLS is more based on the idea that as long as it is not grossly violated, the Central Limit Theorem will save you in large samples. It provides exact inference, if normality is justified. If normality, specifically, is not valid but is not grossly invalid, other methods may be superior, but the cost of a wrong assumption is likely to be small, relative to the fact that it may be too difficult to verify that the errors follow a specific distribution.

Answer 4

Among all the distributions with the given expectation and variance, the normal one is the one with the highest entropy. Therefore, by assuming normality we impose the fewest restrictions on the error distribution in our models, which allows them to be be as data-driven as possible. A more restricted distribution family would mathematically incorporate certain prior beliefs about the processes or phenomena being modeled, limiting the capability of the models to discover information in the data.

Another reason is that the error of your model may be due to the combined effects of a number of unknown and/or unaccounted for factors. If your model is acceptably good, then it will be reasonable to expect that each factor's contribution to the error will be more or less the same. Therefore, if you treat the factors as random variables, you can consider them as iid. Under the central limit theorem their sum will tend to be normally distributed. Since many processes and phenomena in nature and human society are highly complex, it's not unreasonable to assume that the number of factors missing from your model and contributing to its error is fairly large. So, however those random variables are distributed, normal distribution should be a fairly good approximation of their sum's true distribution.

From a more practical point of view, the normality assumption proved to work well in many applications, so it's not that surprising that we came to assume it by default regardless of theoretical and more abstract justifications.