What have to be normally distributed: groups or whole sample?

Question

It is a general and maybe stupid question, but it will help me to avoid a mistake. To test the dependence of continuous variable on nominal with parametric test (t-test, ANOVA), "the data has to be normally distributed" (distribution have to be not much violated from normal).

But I don't clearly understand: I have to test the normality of the continuous data together (whole sample) or the normality of data in every single group?

And if my groups have small sizes (n=4-6), and therefore I fail to reject null, does this indicate a permission to use parametric methods?

Can anyone provide me a link to some good examples of statistical (biomedical) hypothesis testing with R (google did not helped)?

Answer 1

Generally it is the residuals that need to be normally distributed. This implies that each group is normally distributed, but you can do the diagnostics on the residuals (values minus group mean) as a whole rather than group by group. It is possible (and even common) that the data will be approximately normal within each group, but since the group means differ the overall dataset will be quite non-normal, but you can still use normal theory tests for this case.

Note that the real question is not "exactly normal", but rather "normal enough for the given problem". With small datasets the question of normality is the most important, but you have low power to detect non-normality (unless it is very extreeme), with large datasets the Central Limit Theorem kicks in so your data does not need to be that normal, but you have high power to detect small departures from normality. So when doing formal tests of normality as a condition for doing t-tests or anova you are either in the situation where you have a meaningless answer to a meaningful question, or you have a meaningful answer to a meaningless question (there may be some middle size where both are meaningful, but I expect that the middle range is really where both are meaningless).

So, no just because a small sample size does not reject the null does not mean that it is safe to use normal theory methods. Knowledge about the source of the data and some diagnostic plots are likely to be more useful in that decision, or if you are worried about non-normality just go straight to the non-parametric tests.

If you really feel the need for a p-value testing exact normality then you can use the SnowsPenultimateNormalityTest function in the TeachingDemos package for R (but be sure to read the help page).

Another option for testing "normal enough" if you need more than the diagnostic plots is to use the methodology in:

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

(the vis.test function in the TeachingDemos package of R is one implementation of this).

The impartant thing to take away is that knowledge about the process that produced your data is much more important than the output from some program/algorythm written by someone who knows/knew much less about your data and question than you do.

Answer 2

@GregSnow has a really excellent answer here, which I would like to augment with 2 small points. First, while he's right that only the normality of the residuals matters, we can ask the question 'matters for what'? Everyone who has taken stats 101 leaves with the impression that normality is crucial, and that's not really true. The Gauss-Markov theorem tells us that our data do not have to be normal; they simply should be symmetrical and homogeneous. The normality of the residuals is only relevant for the confidence intervals / p-values to be accurate, and even then, only really necessary when sample sizes are small (as @GregSnow points out). When homogeneity of variance obtains, the estimates will be unbiased and efficient whether the residuals are normally distributed or not. My second small addition is that non-parametric tests may not be required with small samples and non-normal residuals--bootstrapping is also a viable option.