2
$\begingroup$

It's surprisingly difficult to find a clear answer to this online. Nearly every online source claims that error terms must follow a normal distribution for statistical inference but stops short of proving it. There are a million "data science" and "statistics" articles out there that rehash the same assertions and simple arguments but neither show any math nor link to any sources.

What concerns me is that I don't see this assumption being used at all in statistical papers, such as "A Practitioner's Guide to Cluster-Robust Inference" by Cameron and Miller (link). In fact, reading through this briefly, I feel increasingly convinced that normality is completely unimportant, except possibly for small sample sizes. As far as I can tell, all that's necessary is for the sampling distribution of the estimated parameter to be approximately normal, and this should be true due to the central limit theorem. Once that is known, then the estimate for the parameter's variance can be used for inference, and this only depends on the variance of the error term and not on its distribution's shape.

The problem is that I'm not a statistician, and I don't have complete confidence in my reasoning. For example, I don't know for sure that CLT results in the parameter's sampling distribution being approximately normal, and I don't know for sure that tests using p-values computed this way don't reject the null hypothesis too often.

$\endgroup$
3
  • $\begingroup$ Simulation is a great way to understand this issue. $\endgroup$ May 20 at 10:24
  • 2
    $\begingroup$ Normal errors aren't necessary for OLS, but they allow a precise statement about the sampling distribution in small samples. $\endgroup$
    – Durden
    May 20 at 12:59
  • 1
    $\begingroup$ OLS doesn't require anything about the errors until you start testing hypotheses. There, what matters are the distributions of the test statistics, not the errors -- and, as in any other hypothesis testing setting, we only need to approximate those distributions adequately. Thus, some applications of OLS couldn't care less about Normality of the errors and others do care, but only to the extent of examining evidence concerning the extent to which possible non-Normality might call into question the p-values being calculated. $\endgroup$
    – whuber
    May 20 at 15:41

1 Answer 1

1
$\begingroup$

BRIEFLY, the Gaussian errors are what guarantee the theoretical sampling distributions, but there is often (but not always) good robustness to violations of this Gaussian ideal.

When you have $iid$ Gaussian error terms, you are guaranteed to have the proper sampling distribution. When you do not, you do not have the proper sampling distribution. In that regard, you do need Gaussian errors.

However, OLS is fairly robust to violations of the Gaussian ideal, and appealing to convergence theorems starts to explain why (more to it than CLT, since the variance is usually unknown, but CLT is a starting point). In that regard, you do not need the Gaussian ideal, since the robustness, especially with large sample sizes, gets you pretty close to the theoretical distribution.

There are, however, error distributions so awful that this robustness is not enough. In that regard, you might not get close to the theoretical distributions without the Gaussian ideal. Further, all of this talk about being “close” lacks the exactness present in so much of mathematics, and what is close for one scientist might not be for another.

$\endgroup$
1
  • $\begingroup$ Thank you for your answer! Do you know of any good sources to learn more about the robustness of OLS, for someone with a mathematical background? $\endgroup$
    – Timoffex
    May 22 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.