I am currently trying to perform a hypothesis test on the difference between four means. Initially I was trying to use ANOVA but then realised I may not meet the assumptions for this test and may need to use the non-parametric version Kruskal-Wallis. I started trying to test the ANOVA assumptions and found that each group had the same variance but using the Shapiro Test I found that my groups were not normally distributed (my sample sizes are small so as far as I'm aware the fact that the test could detect differences in normality suggests there are significant differences from the Normal distribution). I originally concluded that this meant I should use Kruskal-Wallis however looking back at the assumptions for the ANOVA test I realised it actually says to check the normality of the residuals. I'm a little confused by this as I thought not having the assumption of being normally distributed automatically meant we needed to use a non-parametric test. Can someone please explain to me if I do also need to check the normality of residuals and if I do why I need to and how would I check this?
It is important to check for residuals rather than normality of the collection of all responses.
Mixture of normal observations need not be normal. I will give an illustration with $g = 4$ groups and $n = 10$ replications in each group. Data are simulated as normal with several different means and equal variances, but a Shapiro-Wilk test rejects normality for the $gn = 40$ observations taken together.
set.seed(1234) g = 4; n = 10 x1 = rnorm(10, 20, 5); x2 = rnorm(10, 25, 5) x3 = rnorm(10, 35, 5); x4 = rnorm(10, 50, 5) x = c(x1, x2, x3, x4) shapiro.test(x) Shapiro-Wilk normality test data: x W = 0.93777, p-value = 0.0291
Taken together the 40 observations have a normal mixture distribution, which need not be normal. Perhaps Wikipedia on mixture distributions, especially the figure near the top of the page.
Looking at residuals. However, for this simple model, the residuals are found by subtracting the mean for each group from each observation in the group. The 40 residuals pass the Shapiro-Wilk test.
r1 = x1 - mean(x1); r2 = x2 - mean(x2) r3 = x3 - mean(x3); r4 = x4 - mean(x4) r = c(r1, r2, r3, r4) shapiro.test(r) Shapiro-Wilk normality test data: r W = 0.98231, p-value = 0.7743
ANOVA Significant. Because the group population means are quite different, a one-way ANOVA on my fake data shows a highly significant effect.
gp = as.factor(rep(1:4, each=10)) lm.out = lm(x ~ gp); anova(lm.out) Analysis of Variance Table Response: x Df Sum Sq Mean Sq F value Pr(>F) gp 3 5655.9 1885.31 62.167 2.596e-14 *** Residuals 36 1091.8 30.33 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
It is precisely in the cases where there is a significant effect that the aggregated data from all groups are likely to fail the Shapiro-Wilk normality test.
One way ANOVA assumes the residuals are approximately normally distributed around their respective group means. It's not clear above when you ran the Shapiro wilk whether you ran it on the combined raw y values or separately for the y values in each group but neither is correct. In the former case we expect combined scores to become less normal as means become more different. In the latter case you are running multiple tests were you should run only one.
The correct approach is to test the normality of the combined group residuals (yi-group mean). You can use Shapiro or a qqplot for this. If results are not normal consider transforming your raw scores and retesting the residual distribution. Good luck.