1
$\begingroup$

I frequently encounter people saying that normality is assumed in the "errors" in linear regression. It seems what they mean is "residuals" rather than errors. For example, for MLE, we can arrive at the same result as OLS if we assume the residuals are normally distributed zero mean.

I understand that to arrive at the OLS closed form solution, no assumptions about the distribution of the residuals are needed. However, sometimes we make assumptions about the probabilistic distribution of the residuals for statistical inference, e.g., building confidence intervals on our estimators.

Now my question is, when do we typically assume normality in the unobserved errors, and what insight can we gain from this assumption?

Improve this question
$\endgroup$
3
  • $\begingroup$ errors are always unobserved, and for inference they don't need to be normal in OLS. normality is sometimes assumed on errors, not residuals. inference can be done without distributional assumptions on errors because it just happens so that CLT can be applied $\endgroup$
    – Aksakal
    Jul 7, 2020 at 2:52
  • $\begingroup$ @Aksakal Hmm, looks like there's something wrong with my understanding. To arrive at the same result as OLS using MLE, you do assume normality of residuals in that case right? $\endgroup$
    – user5965026
    Jul 7, 2020 at 2:54
  • $\begingroup$ yes, in MLE you have the distribution assumption, and it is on errors, not residuals. only when you assume normal errors you get to the same result as OLS. however, for inference within OLS itself you don't need normal errors $\endgroup$
    – Aksakal
    Jul 7, 2020 at 2:56

1 Answer 1

Reset to default
1
$\begingroup$

Inference about a coefficient value in simple linear regression is based on the ratio of the coefficient estimate $\hat\beta$ to the standard error of the estimate, $s_{\hat\beta}$. See for example this Wikipedia page on simple linear regression with one predictor variable. That ratio is the statistic on which a test against a distribution is made.

The classic normality assumption for inference in linear regression is about the errors, not the residuals. As the Wikipedia page explains, that assumption about the errors* leads to defined distributions of both the numerator and the denominator in that ratio.

Under the assumption of a normal error distribution, $\hat\beta$ has a normal distribution with variance related to the underlying error variance $\sigma^2$. As the residuals result from a normal distribution of errors, the sum of squared residuals used to determine $s_{\hat\beta}$ is "distributed proportionally to $\chi^2$ with $n − 2$ degrees of freedom" for a simple linear regression that estimates an intercept and one slope.

The numerator and the denominator are then independently distributed with their ratio taking a t distribution. So the classic normality assumption is used not directly in the test procedure for linear regression but in the derivation of the t-test.

If the number of observations is large then the law of large numbers and the central limit theorem hold. Then the ratio $\hat\beta/s_{\hat\beta}$, as a good approximation, does behave as a normal distribution. But this is an assumption about the distribution of the test statistic, not about the underlying errors or residuals (except that the errors have finite mean and variance).

This latter case is one example an asymptotic test that holds in the limit of large numbers of observations. I don't know if much can be said about what "insight can we gain from this [normality] assumption" about "unobserved errors," but very often if you see a statistic tested against a normal distribution you will be seeing the application of an asymptotic test.

*People do check the distribution of the residuals to see whether the assumption of normally distributed errors is reasonable, but as this discussion points out such checking can be of limited usefulness.

Improve this answer
$\endgroup$
11
  • $\begingroup$ It looks like I've been thinking about this the wrong way. I had thought the assumption was on the residuals, but when I think about it, that actually makes no sense. The residuals are KNOWN, so no need for assumptions. $\endgroup$
    – user5965026
    Jul 7, 2020 at 3:50
  • $\begingroup$ I don't quite understand the claim that "residuals are drawn from the normal distribution of errors." Doesn't the residual equal the sum of (1) unobserved error / random noise (2) bias error (3) variance error? If so, the bias and variance doesn't affect change the distribution of the residuals? $\endgroup$
    – user5965026
    Jul 7, 2020 at 3:52
  • $\begingroup$ @user5965026 My initial terminology "drawn from" was perhaps imprecise. The residuals represent the result of error around the true regression line (often assumed to be normally distributed errors) plus the uncertainty in the estimated regression line itself (with normal distributions of coefficient values around their true values). I've edited with that in mind. $\endgroup$
    – EdM
    Jul 7, 2020 at 12:25
  • 1
    $\begingroup$ Under classic assumptions (less stringent than the assumption about normality of errors), ordinary least squares regression provides the best linear unbiased estimate (BLUE) of the model. Without bias in the expectation over repeated sampling and modeling, you are left with variance. Residuals from any one model, as errors around the estimated regression line, sum to 0 so they are linearly dependent, unlike errors that are assumed uncorrelated in the proof of the BLUE characterization. $\endgroup$
    – EdM
    Jul 7, 2020 at 12:36
  • $\begingroup$ @user5965026: Think of the residuals as the estimates of the errors ( I called the error the noise term because error confuses me ) except as EdM said, they are linearly independent. Note though that, without the assumption of normality of the noise term, you can't do inference but you also can't just do MLE because minimizing the sum of the noise terms squared in the non-normal case won't necessarily minimize the two normed distance between $Y$ and $X \beta$. The latter is why you have all the GLM machinery. $\endgroup$
    – mlofton
    Jul 7, 2020 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.