2
$\begingroup$

Let us consider the simple linear model $y = \beta_0 + \beta_1 X + \epsilon$, where $y$ is real number, $X$ a matrix of reals and $\epsilon$ is the random "noise". The least-square estimate of the coefficients $\beta_0$ and $\beta_1$ does not depend on the nature of the "noise" term $\epsilon$. It is reasonable to assume that the expectation of $\epsilon$ is zero. For the regression problem really computes $E(y|X)$. It is also reasonable to require that $\epsilon$ has the same variance across the range of $X$. Why do we also require $\epsilon$ to be normally distributed? What does wrong it is not normally distributed but has a distribution symmetric around zero?

$\endgroup$
1
$\begingroup$

The normal distribution is fully defined by its mean and variance, while other distributions are not. Therefore, when residuals are centered and with fixed variance but not gaussian, you can end up with some tricky situations.

Example 1: Skewed residuals
In this example residuals come from a lognormal distribution, and I normalized them to mean 0 and std 1. It is quite evident that this is NOT a good regression model.
enter image description here

Example 2: Bimodal residuals (high kurtosis)
Here I sampled the residuals from a beta distribution. Again, they are centered around 0 and with variance 1 and yet, this is a very bad model.
enter image description here

So, all and all, you would have to fix mean and variance, but also skew and kurtosis, which means 4 total moments. I imagine also further moments can affect the goodness of fit of a regression, so in the end, the solution is fixing just two moments and asking for normality. Of course, the more moments you fix, the closer you're getting to a normal distribution, as your moment generating funciton will be getting more and more similar to a normal's.

EDIT:
As pointed out by @peuhp, the reason is that in those examples we are not maximising likelyhood when doing OLS.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Can you please elaborate more on "the more moments you fix, the closer you're getting to a normal distribution,as your moment generating funciton will be getting more and more similar to a normal's."? $\endgroup$ – Amey Joshi Dec 20 '19 at 9:56
  • $\begingroup$ What I am saying is that my counter examples only shows that you need to have the first four moments to match the normal's (zero mean, fixed variance, zero skew, zero kurtosis). I am quite sure you can find some more counterexamples with some further moments, but as you set more and more moments to 0 to match the normal's (all the ones after the 2nd are 0 for a Gaussian), your characteristic function will become closer and closer to a normal's, converging in distribution. $\endgroup$ – Davide ND Dec 20 '19 at 10:26
  • $\begingroup$ @DavideDN "The reason is that the normal distribution is fully defined by its mean and variance, while other distributions are not. " This does not look at good reason to me : i) other distributions are fully defined by these two parameters, ii) second parametrization is not related to OLS. $\endgroup$ – peuhp Dec 20 '19 at 10:28
  • $\begingroup$ @peuph - The second distribution is perfectly pertinent, it's a linear function with added noise with 0 mean and variance 1, coming from a beta distribution. Shows quite clearly why only fixing the first two moments is not enough. $\endgroup$ – Davide ND Dec 20 '19 at 10:35
  • $\begingroup$ I changed the params of the beta so it's a little less split, hope it helps show my point.. $\endgroup$ – Davide ND Dec 20 '19 at 10:39
3
$\begingroup$

Why do we also require ϵ to be normally distributed?

IID normally distributed $\epsilon$ are needed to ensure that the parameters $\beta_0$ and $\beta_1$ provided by ordinary least square corresponds to the maximum likelihood estimators. If it is not the case, you don't have a guaranty of optimality of the OSL solutions.

So basically, if noise is not gaussian you are using the optimal solution of a problem to solve another problem ; which may cause some troubles.

Actually, if the noise is considered symmetric wrt 0 and iid (but not necessarly gaussian) the Gauss-Markov theorem ensures OSL is the best linear unbiased estimator. However, there is no reason to limit ourself to linear estimator.

The answer given by @Davide DN give you some illustrations about what may happen in practice. Another illustrating example is when using a uniform noise on [-1,1] (so symmetric), where you can easily build settings where the OLS provides solution corresponding to a probability 0.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Definitely a better answer, I just gave a counter example on why it is not enough to ask for exogeneity and spherical errors :) $\endgroup$ – Davide ND Dec 20 '19 at 10:51
  • $\begingroup$ @DavideDN Well, our answers completement each others. However, I really think your first sentence "The reason is that the normal distribution is fully defined by its mean and variance, while other distributions are not. " is missleading and think it should be removed/repharse $\endgroup$ – peuhp Dec 20 '19 at 10:54
  • $\begingroup$ You're right, I removed the "the reason is". Definetly misleading since I was giving an example and not a full explanation! $\endgroup$ – Davide ND Dec 20 '19 at 11:00
1
$\begingroup$

The assumptions of OLS are vague, and they only ensure optimality within a very restrictive set of estimators, specifically those which are linear estimators of the mean of a distribution. What this basically boils down to is that the best way to construct a weighted average for estimating a mean is to choose weights that are inversely proportional to the variance. This is a very restrictive consideration that does not consider, for example, the possibility that the median may also be a valid measure of location, since it is a nonlinear estimator. The reason normality is often assumed is that if you are trying to estimate the mean of the distribution, then certain general inferential practices are justified.

First, if the distribution is normal, estimation and inference based on F-tests and t-tests are valid for any sample size. Second, if the distribution is not normal, but it is symmetric, unimodal, and has finite constant variance with a reasonably large support, then for sufficiently large samples, inferences based on t-tests and F-tests will be approximately valid due to the Central Limit Theorem.

If the mean of the distribution is not a reasonable measure of location, then OLS is not a reasonable method of estimation. For example, if the errors are either bimodal or skewed, one would not use the mean to summarize such a distribution.

If the errors are such that the mean is reasonable, it may well be that another estimator that is nonlinear could be an improvement. For example, if the errors follow a Laplace Distribution, then the MLE (sample median) is an improvement over OLS. However, in practice, it can be difficult to distinguish errors that follow a Laplace Distribution from those which follow another distribution that has similar properties. Since, in these situations, there may not be a large price to incorrectly assuming normality if the sample size is sufficiently large, then the OLS estimation with normality assumptions will likely produce valid conclusions anyway.

So, to summarize, the main reasoning for normality assumptions in OLS is more based on the idea that as long as it is not grossly violated, the Central Limit Theorem will save you in large samples. It provides exact inference, if normality is justified. If normality, specifically, is not valid but is not grossly invalid, other methods may be superior, but the cost of a wrong assumption is likely to be small, relative to the fact that it may be too difficult to verify that the errors follow a specific distribution.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Among all the distributions with the given expectation and variance, the normal one is the one with the highest entropy. Therefore, by assuming normality we impose the fewest restrictions on the error distribution in our models, which allows them to be be as data-driven as possible. A more restricted distribution family would mathematically incorporate certain prior beliefs about the processes or phenomena being modeled, limiting the capability of the models to discover information in the data.

Another reason is that the error of your model may be due to the combined effects of a number of unknown and/or unaccounted for factors. If your model is acceptably good, then it will be reasonable to expect that each factor's contribution to the error will be more or less the same. Therefore, if you treat the factors as random variables, you can consider them as iid. Under the central limit theorem their sum will tend to be normally distributed. Since many processes and phenomena in nature and human society are highly complex, it's not unreasonable to assume that the number of factors missing from your model and contributing to its error is fairly large. So, however those random variables are distributed, normal distribution should be a fairly good approximation of their sum's true distribution.

From a more practical point of view, the normality assumption proved to work well in many applications, so it's not that surprising that we came to assume it by default regardless of theoretical and more abstract justifications.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, Milos. I had not come across the points in your first and second paragraphs. $\endgroup$ – Amey Joshi Dec 20 '19 at 16:26
  • $\begingroup$ You're welcome, @AmeyJoshi. The basic idea of the first one is usually mentioned in information theory textbooks, whereas the other one can be found in some books on statistics, but not all. I would have pointed you to the literature, but don't recall where I read them exactly. $\endgroup$ – Milos Dec 21 '19 at 1:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.