3
$\begingroup$

In data, what causes the error term to be non-normally distributed in regression? Along the same lines, what solutions are there for non-normal residuals?

For example, is it caused solely by a non-normally distributed dependent variable? Does the distribution of independent variables play a role?

Extending that, would we expect the error term to become normally distributed when we include/exclude different independent variables? Practically, I'm not sure how to decide what variables to include if I don't know how they affect my ability to assess model fit.

$\endgroup$
  • $\begingroup$ Under the standard assumptions, error terms are normally distributed. What assumptions, then, do you have in mind? And by "causes" are you asking about physical causes in the data generation process or about the mathematical effects of the OLS estimation? $\endgroup$ – whuber Oct 13 '14 at 18:57
  • 1
    $\begingroup$ Edited to clear up that I mean causes in the DGP $\endgroup$ – user7340 Oct 13 '14 at 19:09
  • $\begingroup$ You might be conflating errors with residuals. If you misspecify a model where the observations actually have normal errors, you might see distinctly non-normal residuals. The actual (unobservable) errors, $\varepsilon\,$, don't know what your model is, they're just doing their own thing, making noise. You just get really biased estimates of them, that won't look anything like the thing they're supposed to represent, because you got the model wrong. (Edit: now I've seen the answers, I see that this possible cause of confusion seems to have led to different interpretations of your question) $\endgroup$ – Glen_b Oct 13 '14 at 22:49
5
$\begingroup$

It's an interesting question. Flippantly, it's a bit like saying "Why are tigers not lions?" to which the entirely accurate but supremely unhelpful answer is "Because they are tigers, not lions", but it naturally deserves a more serious answer.

Any answer which is, directly or indirectly, an appeal to how convenient or simple it would be if error terms were normal or Gaussian is naturally just wishful thinking, or the appeal of nice things being nicer than nasty things.

The question one way or another involves a large fraction of statistical science, but no answer can cover it all.

First I flag that to me there's a distinction:

(a) what the error term is "out there"

(b) what error term is postulated in the model

(c) what the residuals are when calculated from data.

The distinction between (b) and (c) is surely standard; mentioning (a) may well seem more dubious or at least less standard, but I think it's needed, as ideas about (a) should be behind hypotheses on (b).

Also, I suggest that a general answer to the question can't be based on assuming that error is always additive to some deterministic part, as is explicit in linear regression models. So, the error term to me implies the structure of error taken generally.

Further, I am not referring to residuals directly. I am focusing on (a) with the intended message that (a) should imply (b). Terminology is a beast here as always: the literature I read talks about residuals only as quantities calculated from the data.

If you focus on why an error term might be normally distributed, the simplest answer is because you got the deterministic part of a model almost exactly right and everything else is essentially lots of little things which, by the central limit theorem, when combined should be normal or Gaussian as a good approximation. Historically one root of this kind of model is in astronomy where often, but not always, to a very good approximation the errors are just small measurement errors.

However, there are plenty of situations in which "everything else" does not follow that description. You'll get different views across statistically-minded people on how common that is. It's arguable that the prominence of linear models with Gaussian error terms is just a kind of historical hangover hinging on various accidents: that this was the first kind of method to be worked out in real detail; that it works when it works, more or less; and that applications of this model were often relatively easy with what now are primitive calculation methods preceding the electronic computer. Also, people have invented all sorts of trickery for bending or extending the linear model in any case. Within statistical science at present, econometrics perhaps represents the extreme view that models of this kind remain central, although to be fair econometricians have been as active as any other group in exploring alternatives.

For exceptions, I will mention just two. For binary responses, with possible values say 0 or 1, such as present or absent, survived or not, and so on and so forth, the stochastic part of a model cannot be normal even in principle. For non-negative counts or other responses, the same can hold true, and the starting point is more likely to be a Poisson or some other non-normal distribution.

P.S. Whether you are using ordinary least squares has itself no influence on whether error terms are normally distributed. Naturally, it is true that if they are, then OLS is appealing as an estimation method, but that's not at all the same.

EDIT: Thanks to @whuber for his firm but gentle encouragement to clarify as far as possible. The question has morphed since I first posted, but surely (and benignly) allows answers of quite different styles.

$\endgroup$
  • $\begingroup$ Re the PS: Are you really claiming that the estimation method does not affect the residuals? I think counterexamples should be easy to come by :-). That leads me to understand that you are using "error term" in the sense of the model and not in the sense of residuals from the OLS fit. But that raises even thornier questions: if a Gaussian linear model is the one you adopt, then by assumption its "error terms" are Normal. Why doesn't that make the assertion "error terms are Normal" a mere tautology? $\endgroup$ – whuber Oct 13 '14 at 19:34
  • $\begingroup$ Not at all. To me there's a distinction: (a) what the error term is "out there" (b) what error term is postulated in the model (c) what the residuals are when calculated from data. The distinction between (b) and (c) is surely standard; mentioning (a) may well seem suppositious, but I think it's needed, as ideas about (a) should be behind hypotheses on (b). Also, I suggest that a general answer to the question can't be based on assuming that error is always additive to some deterministic part. So, the error term to me implies the structure of error. (The question has morphed since I posted.) $\endgroup$ – Nick Cox Oct 13 '14 at 19:45
  • $\begingroup$ Thanks, but I'm still not sure which of (a), (b), or (c) you are using in your answer! $\endgroup$ – whuber Oct 13 '14 at 20:16
  • 1
    $\begingroup$ I am not referring to residuals at all. I am focusing on (a) with the intended implication that (a) should implying (b). Terminology is a beast here as always: the literature I read talks about residuals only as quantities calculated from the data. $\endgroup$ – Nick Cox Oct 13 '14 at 20:31
  • $\begingroup$ Thank you. Now that @PeterFlom has weighed in with an implicit interpretion of "error term" as synonymous with residual, I will leave it to the two of you to discuss which interpretation is the appropriate one! Regardless, in order to be properly understood, I would appeal to both of you to edit your posts to make plain which interpretations you are adopting in your answers. $\endgroup$ – whuber Oct 13 '14 at 21:37
2
$\begingroup$

Since one possible cause of non-normal residuals is a missing variable, one possible cure is to include that variable (or a good proxy). But that isn't the only possible cause.

The dependent variable need not be normally distributed for the errors (as measured by the residuals) to be normal. For instance if you have a regression of adult human height on sex, the DV (height) would be bimodal but the residuals would be very close to normal. If you used height in all humans on age and sex, then the DV would be skew and probably bimodal, but the residuals would be very close to normal

If you have non-normal residuals, you can use methods that don't rely on that assumption - one such is quantile regression.

EDIT in response to @whuber 's comment.

In OLS regression, it is assumed (among other things) that the errors are normally distributed. We can't test this directly as we can't look at the errors. So we look at the residuals and see if they are roughly normal. Usual ways of doing this include quantile normal plots.

$\endgroup$
  • $\begingroup$ This needs clarification, especially in light of Nick Cox's existing answer which interprets "error term" differently. Residuals--as realized in any particular dataset as a finite set of numbers--cannot be Normal. Are you thus referring to the theoretical ones, considered as random variables? But then any individual residual still has a Normal distribution even when variables are omitted. Therefore you must be thinking in terms of (a) the realized residuals as (b) lumped together in one undifferentiated collection and (c) compared to a Normal distribution for reference, right? $\endgroup$ – whuber Oct 13 '14 at 21:40
  • $\begingroup$ I am referring to one of the basic assumptions of OLS regression, that the errors are normally distributed. But of course we can't know the errors, so we use the residuals. And, to test the assumptions, we often use things like quantile normal plots to see if the residuals are normally distributed. It doesn't really seem to need clarification, to me, unless one is trying to misunderstand it. $\endgroup$ – Peter Flom Oct 13 '14 at 22:51
  • $\begingroup$ The comment helps, thank you. But accusing me of willful misunderstanding is just evading the point: the question is ambiguous and so, therefore, is your answer until you edit it to be more explicit about your meaning--especially when it appears to be so different from a previous answer. Just because you understand what you are talking about does not mean that all our readers will share that understanding. $\endgroup$ – whuber Oct 13 '14 at 23:21
  • $\begingroup$ OK, I guess if you think it could be ambiguous, it could be. I haven't run into that before when I say things like this to my clients, but maybe it takes a greater degree of sophistication to see the ambiguity. $\endgroup$ – Peter Flom Oct 14 '14 at 0:10
  • 1
    $\begingroup$ To my knowledge normally distributed errors is not a basic assumption of OLS regression (although many undergrad. textbooks do this). Asymptotic normality of the OLS estimator is a result of the CLT not by assuming normality of the error term. The OLS estimator will be consistent even if we do not have normally distributed errors. This is among others why OLS is more robust than MLE. We do not need any distributional assumptions. $\endgroup$ – Plissken Oct 14 '14 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.