8
$\begingroup$

I've read once that the normality assumption shouldn't be a problem and that you actually shouldn't care that much if your sample is large. Why is that? Can someone give me a mathematical explanation? And is that for both prediction and inference (i.e getting BLUE estimators)?

$\endgroup$
3
  • 3
    $\begingroup$ Research the Central Limit Theorem $\endgroup$
    – whuber
    Feb 25 at 16:41
  • 3
    $\begingroup$ For prediction you don't even need normality; the Gauss-Markov theorem says that as long as you have independence, equivariance (homoscedasticity), and mean-zero errors, then the OLS estimates will be BLUE. $\endgroup$
    – Ben Bolker
    Feb 25 at 16:47
  • 1
    $\begingroup$ Yes, they will be BLUE but that of itself doesn't give you the large sample type I error rate on the hypothesis tests, nor the coverage of CIs, which the normal assumption was certainly used in the derivation of. To argue that these properties work in large samples without that assumption you need to invoke something related to a form of CLT. $\endgroup$
    – Glen_b
    Feb 26 at 1:23

2 Answers 2

16
$\begingroup$

The assumption of the Normality of the error term in a regression that applies Least-Squares estimation methods, is used to make statistical inferences about the coefficients after estimation, it is not used in the estimation itself.

Under Normality of the error, the LS estimator of the coefficients follows itself a Normal distribution already in a finite sample (of any size). So we can conduct inference like t-tests and the like, and they will be based on the exact finite-sample distribution.

If we do not have Normality of the errors, then the LS estimator has in general unknown finite-sample distribution but still has (under some of the other assumptions characterizing the model) a Normal distribution asymptotically / at the limit. And the larger the sample size, the more closer to the asymptotic distribution will the true unknown finite-sample distribution of the LS estimator be.

So the statistical inference we will conduct without Normality of the error while we will treat it as if it was Normal, will be approximate ("asymptotically valid"), but with "large" sample sizes, we anticipate that it will not be misleading.

$\endgroup$
5
$\begingroup$

And is that for both prediction and inference (i.e getting BLUE estimators)?

No, not for both

  • The estimator for the coefficients will approach a normal distribution, independent of the distribution of the errors $\epsilon$ (with the condition that they are independent, have zero mean and finite variance $\sigma^2$)

    $$\hat{\beta} \to N(\beta,\Sigma) \qquad \text{with $\Sigma = \sigma^2 (X^TX)^{-1}$}$$

    An heuristical argument is that it is because the parameter estimates $\hat{\beta}_j$ are a linear sum of the observations a linear sum of the observations $y_i$ (and sums tend to approach a normal distribution, independent of the distribution of the terms)

    $$\hat\beta_j = \sum_{i=1}^n a_{ji}y_i \qquad \text{with} \, a_{ji} = \left[(X^TX)^{-1}X^T\right]_{ji}$$

    You may find more about it in several questions here. E.g. search with keywords OLS large sample

  • The predictions will include the error of the estimate plus the error from a single observation which will still relate to the error distribution $\epsilon$. For large samples, there is still no approach to a normal distribution here.

    Example, say you have a population that follows an exponential distribution with unknown rate $\lambda$. Based on a large set of observations $y_1,y_2,\dots, y_n$ you estimate the rate $\hat\lambda$ and now you wish to compute a prediction interval for a next observation $y_{n+1}$. To do this, you will use $$y_{n+1} \sim Exp(\hat\lambda)$$ and it would be wrong to assume normality $$y_{n+1} \sim N(1/\hat\lambda,1/\hat\lambda^2)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.