3
$\begingroup$

One of the assumptions of ANOVA is normally distributed residuals. However, as noted here some authors have stated that this assumption is barely important at all. Simulation results have shown that, provided sufficient sample size, deviations from normality can be tolerated without a large adverse effect. In practice I often hear people recite that 'the test is robust to violations of the assumption of normality'.

My question is this: When making inferences at the group level (not point estimates for individual observations), why is it not the case that the assumption is about the normality of the error distribution (i.e. the hypothetical distribution that we quantify with terms like the Standard Error of the Mean) as opposed the specifically observed residuals?

Isn't the most proximal cause of Type I inflation or deflation, non-symmetrical errors in estimating the mean, etc, really a reflection of the non-normality of the error distribution?

I grant that, clearly, there is an equivalence here that may lead to this being a bit of semantics. Non-normally distributed residuals almost inevitability are going to lead to a somewhat non-normal error distribution of the mean estimate. However, I think the simulation results I referred to above (and our general knowledge of the Central Limit Theorem) hold out the idea that if the means being described have a sufficient number of samples, then the error distribution can become reasonably normal and thus nothing bad happens.

Moreover, if the assumption was about the distribution of errors it would neatly describe why it is that the violation of this assumption can frequently be papered over with larger sample sizes. It would also highlight characteristics of datasets where 'larger' is so huge as to make it impractical (e.g. zero inflated datasets).

Is there something I am missing that makes it critical that we describe the assumption as 'normality of residuals'?

Improve this question
$\endgroup$
8
  • $\begingroup$ Possible duplicate: stats.stackexchange.com/questions/60410/… $\endgroup$
    – HStamper
    Commented Jan 4, 2017 at 16:17
  • 1
    $\begingroup$ The assumption is that $y \sim \mathcal{N} \left( X \beta, \sigma^2 I \right) = X \beta + \mathcal{N} \left( 0, \sigma^2 I \right)$, which is on the error distribution. Under this assumption, it follows that the regression's residuals $(I - H) y \sim \mathcal{N} \left( 0, \sigma^2 (I-H) \right)$, where $H = X (X^T X)^{-1} X^T$. They're both normal. However, note that the second distribution does not imply the first. $\endgroup$
    – user795305
    Commented Jan 4, 2017 at 16:27
  • $\begingroup$ @Benjamin Clearly. :) That's the part where 'there is an equivalence here that may lead to this being a bit of semantics' comes in. Thanks for defining that equivalence so concisely. However, I don't think I've found a source that states that the assumption is on the error distribution rather than on the regression residuals. Have you? If you can answer with a source, then the check mark is yours. $\endgroup$
    – russellpierce
    Commented Jan 4, 2017 at 16:33
  • $\begingroup$ That's what my last sentence comments on. The assumption has to be on the distribution of $y$. If we only assumed the distribution of $(I-H) y$, which is strictly weaker than assuming the distribution of $y$, then, for instance, ANOVA tables wouldn't exist: what would the distribution of $Hy$ be? $\endgroup$
    – user795305
    Commented Jan 4, 2017 at 16:38
  • 1
    $\begingroup$ Residuals are observed, so there is no need to make assumptions about them. Meanwhile, errors are unobserved, and assumptions such as normality are common on them. $\endgroup$
    – Richard Hardy
    Commented Jan 5, 2017 at 13:26

1 Answer 1

Reset to default
4
$\begingroup$

The assumption is on the error distribution so that $y \sim \mathcal{N} \left( X \beta, \sigma^2 I \right)$. See, for instance, page 9 of https://www.stat.osu.edu/~pfc/teaching/7410/notes/A1_linear_models.pdf. For a textbook that mentions this, see the first page in chapter 3 of Agresti's Foundations of Linear and Generalised Linear Models--I think the textbook is amazingly clear.

That simulation linked to comments on the robustness of the $F$ test to normality. If, however, we generated data with with mean $X\beta$ and some highly non symmetric error, there'd be issues. You mention perhaps invoking the CLT to straighten out everything, but there are no sums here to invoke the CLT on. We're just accumulating more and more samples from whatever the residual distribution is.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Would you please clarify in lay-terms, when you say 'on the error distribution' do you mean the hypothetical distribution that we quantify with terms like the Standard Error of the Mean, the specific residuals observed, or the hypothetical distribution of residuals one might observe? $\endgroup$
    – russellpierce
    Commented Jan 4, 2017 at 22:35
  • $\begingroup$ There's really only 2 distributions at play here. The standard error of the mean is just a transformation of the sample standard deviation of a random variable. The "specific residuals observed" is the regression's residuals $(I-H)y = y - \hat{y}$ and the "hypothetical residuals" are the residuals if we knew the true regression line, which is $y - X \beta$. Under the assumptions of a linear model (that $y \sim \mathcal{N} \left( X \beta, \sigma^2 I \right)$), we have that the mean is $X \beta$ and the "hypothetical residuals" have distribution $\mathcal{N} \left( 0, \sigma^2 I \right)$. $\endgroup$
    – user795305
    Commented Jan 5, 2017 at 12:48
  • 1
    $\begingroup$ Ah. Thank you. Sorry it took me a while to catch on. Flu + statistics is a bad mix. :) $\endgroup$
    – russellpierce
    Commented Jan 5, 2017 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.