What type of data would have non-normal errors?

Question

I'm trying to understand the assumptions for an OLS model. I get that the error term should be normally distributed if we want easy-to-calculate confidence intervals for our coefficient estimates.

But what type of data would result in non-normal errors?

I'm not sure if that's a cut-and-dried answer or if the answer depends on the model you use (e.g. if the model includes squared terms, etc.). But what I'd love to know if I can look at a set of data BEFORE running OLS and determine if the errors would be non-normal. For example, would this only occur when the data themselves are non-normally distributed (on one or more variables)?

Answer 1

Here are a few examples of data you might easily encounter that you can pretty much conclude on sight won't have normal distributions, even conditionally: Counts, 0/1 data, continuous proportions, times, Likert-scale data, monetary amounts (like incomes, or claim sizes).

But what I'd love to know if I can look at a set of data BEFORE running OLS and determine if the errors would be non-normal.

You can bet they're not exactly normal in any case.

It's not really the right question to ask though (essentially all our models are not exact descriptions, so it's not actually the exercise here).

In large samples normality may be much less important than the other considerations (the assumption of constant variance, or of independence can be quite important and their impact doesn't decrease with increasing sample size)

For example, would this only occur when the data themselves are non-normally distributed (on one or more variables)?

This is the distinction between conditional and marginal distribution. If you just look (say via a Q-Q plot) at the distribution of your response variable (DV) it doesn't tell you whether your residuals (as proxies for your errors) might be reasonably normal; it depends on how the X's are arranged. You can have normal looking residuals but the DV looks distinctly non-normal -- or vice-versa.

Arthur Charpentier gives some discussion of this issue for both linear models and GLMs here: Simple distributions for mixtures.

Answer 2

Non-normal errors are common when the response variable is nonnegative or bounded, which might happen in count data, time-to-event data, or scores on a statistics midterm. It is difficult to tell how badly this assumption fails, though, without fitting the model and looking at the residuals. Of course, if you are willing to choose what type of regression to do (or what transformation to do) by looking at the data, your p-values will not work right.

Of three main assumptions in OLS regression, normality of errors is generally considered least important. More important is that errors should be uncorrelated and have roughly constant variance.

Answer 3

Here are some examples of when you might expect non-normal errors:

1. Insurance - Let's say an insurance company insures people against something (e.g. their home being flooded), then you would expect most people not to claim anything over, say, a year, and many of those who do claim to claim rather a lot (or something far away from your mean). Rather than having most claims close to your mean as you expect with normal errors, you have most below (at 0), and some much higher.
2. Anything where the error can't take any real value - the normal distribution assigns strictly positive probabilities to the whole of the real line, in other words, there's always some chance (albeit very small) that you'll get a value which is really far away from your mean, which may not be possible in practice. For example, if I observed people on the street, and tried saying they have a mean age of 30 with a N(0, 7^2) error, that wouldn't be true, since then my model would suggest that occasionally people on the street would have negative age! In some circumstances, if the variance is small enough and mean far enough away from these values which aren't allowed, a normal distribution may still seem somewhat reasonable.
3. Other distributions are natural - e.g. for the ages of pupils in a classroom, of 12-13 year olds, we might assume we have uniform errors, rather than normal.
4. The normal distribution has light tails - what I mean by this is that as you go away from the mean, the chances of observing data so far out gets exponentially smaller. In some instances, this may be okay, but in others it may not, and if this can't be justified, it will make your model more sensitive to outliers. I'm struggling to think of a particularly clear example for this one, but will return and edit if I think of one soon.
5. Discrete data - we may have discrete errors if we're trying to measure something qualitative e.g. in diagnosing someone for a particular disease.