29
$\begingroup$

Gelman and Hill (2006) write on p46 that:

The regression assumption that is generally least important is that the errors are normally distributed. In fact, for the purpose of estimating the regression line (as compared to predicting individual data points), the assumption of normality is barely important at all. Thus, in contrast to many regression textbooks, we do not recommend diagnostics of the normality of regression residuals.

Gelman and Hill don't seem to explain this point any further.

Are Gelman and Hill correct? If so, then:

  1. Why "barely important at all"? Why is it neither important nor completely irrelevant?

  2. Why is the normality of residuals important when predicting individual data points?

Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
27
$\begingroup$

For estimation normality isn't exactly an assumption, but a major consideration would be efficiency; in many cases a good linear estimator will do fine and in that case (by Gauss-Markov) the LS estimate would be the best of those things-that-would-be-okay. (If your tails are quite heavy, or very light, it may make sense to consider something else)

In the case of tests and CIs, while normality is assumed, it's usually not all that critical (again, as long as tails are not really heavy/light), in that, at least in not-very-small samples the tests and typical CIs tend to have close to their nominal properties (not-too-far from claimed significance level or coverage) and perform well (reasonable power for typical situations or CIs not too much wider than alternatives) - as you move further from the normal case power can be more of an issue, and in that case large samples won't generally improve relative efficiency, so where effect sizes are such that power is middling in a test with relatively good power, it may be very poor for the tests which assume normality.

This tendency to have close to the nominal properties for CIs and significance levels in tests is because of several factors operating together (one of which is the tendency of linear combinations of variables to have close to normal distribution as long as there's lots of values involved and none of them contribute a large fraction of the total variance).

However, in the case of a prediction interval based on the normal assumption, normality is relatively more critical, since the width of the interval is strongly dependent on the distribution of a single value. However, even there, for the most commonly used interval (95% interval, two-sided), the fact that many unimodal distributions have very close to 95% of their distribution within about 2sds of the mean tends to result in fairly reasonable performance of a normal prediction interval even when the distribution isn't normal for a fairly wide variety of distributions. [This doesn't carry over quite so well to much narrower or wider intervals -- say a 50% interval or a 99.9% interval -- though.]

Substantive violations of other assumptions, including linearity of relationship with the supplied predictor(s), constant variance and independence will be more consequential in large samples than normality typically will be.

Improve this answer
$\endgroup$
7
  • $\begingroup$ "The tendency of linear combinations of variables to have close to normal distribution." -- I suppose this is not connected to the Central Limit Theorem. Is it? If not, what kind of "theorem" is this statement? $\endgroup$
    – Heisenberg
    Commented May 17, 2015 at 18:18
  • 1
    $\begingroup$ @Heisenberg It has a connection to particular versions of the CLT, yes. (see the Lyapunov and Lindeberg versions here). If you want a theorem to apply for finite samples, we're looking at a version of the Berry-Esseen theorem. But the statement was intended more an observation (hence the use of the word "tendency") than a theorem. $\endgroup$
    – Glen_b
    Commented May 18, 2015 at 0:06
  • $\begingroup$ More specifically, if the weights in the linear combinations are such that one variance ($a_i^2 Var(Y_i)$) doesn't dominate the rest (in the sense that all these variances vanish as a fraction of the total as $n\to\infty$), then there's a CLT that covers it, but as I said, I wasn't trying to actually invoke the theorem per se. $\endgroup$
    – Glen_b
    Commented Aug 5, 2020 at 8:31
  • $\begingroup$ For CI based off std with no assumption on the distribution there’s always Chebyshev’s inequality. So 2stds will give you 75% confidence and 4 stds will give you close to 95%. Unimodality has little to do with that $\endgroup$
    – SBF
    Commented Jun 12, 2023 at 6:46
  • 1
    $\begingroup$ Alternatively if you make some different assumptions about the data generation process you can also show that the estimators have asymptotic normal distributions: eml.berkeley.edu/~powell/e240b_sp10/alsnotes.pdf $\endgroup$
    – Ben
    Commented Jul 30, 2023 at 21:21
7
$\begingroup$

2: When predicting individual data points, the confidence interval around that prediction assumes that the residuals are normally distributed.

This isn't much different than the general assumption about confidence intervals -- to be valid, we need to understand the distribution, and the most common assumption is normality. For example, a standard confidence interval around a mean works because the distribution of sample means approaches normality, so we can use a z or t distribution

Improve this answer
$\endgroup$
1
  • $\begingroup$ please explain how confidence interval around that prediction is found ? " the confidence interval around that prediction assumes that the residuals are normally distributed." $\endgroup$
    – star
    Commented Feb 14, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.