Generally it is the residuals that need to be normally distributed. This implies that each group is normally distributed, but you can do the diagnostics on the residuals (values minus group mean) as a whole rather than group by group. It is possible (and even common) that the data will be approximately normal within each group, but since the group means differ the overall dataset will be quite non-normal, but you can still use normal theory tests for this case.
Note that the real question is not "exactly normal", but rather "normal enough for the given problem". With small datasets the question of normality is the most important, but you have low power to detect non-normality (unless it is very extreeme), with large datasets the Central Limit Theorem kicks in so your data does not need to be that normal, but you have high power to detect small departures from normality. So when doing formal tests of normality as a condition for doing t-tests or anova you are either in the situation where you have a meaningless answer to a meaningful question, or you have a meaningful answer to a meaningless question (there may be some middle size where both are meaningful, but I expect that the middle range is really where both are meaningless).
So, no just because a small sample size does not reject the null does not mean that it is safe to use normal theory methods. Knowledge about the source of the data and some diagnostic plots are likely to be more useful in that decision, or if you are worried about non-normality just go straight to the non-parametric tests.
If you really feel the need for a p-value testing exact normality then you can use the SnowsPenultimateNormalityTest
function in the TeachingDemos
package for R (but be sure to read the help page).
Another option for testing "normal enough" if you need more than the diagnostic plots is to use the methodology in:
Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
D.F and Wickham, H. (2009) Statistical Inference for exploratory
data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
367, 4361-4383 doi: 10.1098/rsta.2009.0120
(the vis.test
function in the TeachingDemos
package of R
is one implementation of this).
The impartant thing to take away is that knowledge about the process that produced your data is much more important than the output from some program/algorythm written by someone who knows/knew much less about your data and question than you do.