# Contradicting p-values for Anova and Kruskal-Wallis on same data: Which is right?

I have a data file with task duration values for three groups, and I want to determine group effect on task duration (tasks were executed by individuals; each group had 7 different individuals; each individual executed the same three tasks; and the data for one individual in group B was not recorded because of a setup problem during the experiment).

I have created from the data file the following box plot (red dots are the means, and "n" is the number of time values in each group):

and also the following histogram (duration given in "min:sec"):

My data sample per group is small, and "Shapiro-Wilk normality test" tells me that group A does not come from a normal distribution, and that groups B and C come from a normal distribution. Because groups are small and one group is non-normal, I decided to run Kruskal-Wallis one-way analysis of variance (non-parametric) and its result was:

Kruskal-Wallis rank sum test
data:  Duration by Group
Kruskal-Wallis chi-squared = 4.2811, df = 2, p-value = 0.1176


so I should accept that the effect of the groups was not significant (p-value > 0.05).

However, when I run one-way Anova (sanity check just in case Kruskal-Wallis was not the correct choice), Anova's result was:

ANOVA Duration ~ Group
Df    Sum Sq   Mean Sq F value  Pr(>F)
Group        2 0.0003692 1.846e-04   6.473 0.00293 **
Residuals   57 0.0016257 2.852e-05
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Tukey multiple comparisons of means
95% family-wise confidence level

B-A -0.0055763154 -0.009704328 -0.0014483027 0.0054132
C-A -0.0048032407 -0.008769307 -0.0008371744 0.0138750
C-B  0.0007730747 -0.003354938  0.0049010874 0.8943085


That is, Anova returns p-value < 0.05, that is, it is telling that the group effect is significant (in this case, group A was significantly different regarding B and C).

However, because of a small number of samples per group and the fact that one group is not normally distributed, I tend to accept Kruskal-Wallis result, but I am not sure.

So my questions are:

Should I accept Kruskal-Wallis result because this test is more adequate than Anova for this particular case?

How should I have used the box plot and the histogram to decide for the most adequate test?

Thanks

-
The assumption is that the residuals are normally distributed, what is the result of the Shapiro-Wilks test for the residuals? –  Glen Jul 5 '12 at 1:56
@Glen The Shapiro-Wilk test for the residuals of group A is the same as for group A (the residuals where computed as indicated here) –  mljrg Jul 5 '12 at 10:33

The boxplot and histogram tell you all by themselves that your data are skewed, especially in group A. The Shapiro-Wilk test is kind of pointless. With data thusly skewed the ANOVA isn't really appropriate. The Kruskal-Wallis rank sum test is based on the ranks, not the absolute values and doesn't require normality, either in the measures or residuals. It is the more appropriate test.

A quick Google search will tell you one requires normality and one does not.

One thing you might consider is that durations are an arbitrary representation of time. For example, you can indicate the duration of an event as 2s or you can say the event has a rate 0.5 events/s. It's the exact same thing and both numbers can arbitrarily be interchanged for representation. However, rates tend to be much less skewed and more appropriate for statistical analysis. It's possible your rates are normally distributed and you can use ANOVA in that case.

If you do decide to look at rates keep in mind that the direction of magnitude changes, a higher duration values = a lower rate value. Some people use a negative rate just to avoid that confusion.

-
If I did the transformation into ratios correctly, in R the Shapiro-Wilk results are: shapiro.test(1/A): W = 0.9064, p-value = 0.04657 shapiro.test(1/B): W = 0.9026, p-value = 0.06388 shapiro.test(1/C): W = 0.6018, p-value = 2.057e-06 That is, at alpha=0.05, Shapiro-Wilk normality test fails for groups A and C, so Anova is not adequate again ... –  mljrg Jul 5 '12 at 11:50
It's probably not but I already indicated the test isn't the best way to tell, just look at the data. Regardless, you've got a resonable non-parametric solution. To summarize, there is no good "test" for normality because even with very normal data it will always fail if N is high enough. Nevertheless, you have a small N and the test is reflecting what you would see in plots anyway. –  John Jul 5 '12 at 13:26

The Kruskal-Wallis test and the Anova test are testing different hypotheses, both could be correct, the answers differ because they are answering different questions.

-
The tests address slightly different hypothesis and it is clear that the medians are closer than the means. But I think the ANOVA test is still inappropriate to apply to this data and does not provide a correct answer. –  Michael Chernick Jul 5 '12 at 3:45
While the data is clearly skewed, with a sample size of 60 (probably 20 per group) the CLT may give a good enough approximation (multiply the p-value by 10 and it will still be significant at the 0.05 level). It would be interesting to look at permutation and/or bootstrap tests comparing the means and the medians of the groups (even though the KW test is not technically a test of medians either). –  Greg Snow Jul 5 '12 at 4:04
@GregSnow A is highly skewed because one task took much more time (the values above 21 min) than the others in A. A's effect surely makes individuals take more time in that task compared with the same task in B and C, in which individuals did not do the extra work caused by the effect of group A. However, as Kruskal-Wallis compares medians (centrality), and the durations for all other tasks are close among groups, I tend to accept the result of Kruskall-Wallis, especially as the extra time (bug fix) for the task in group A would be spent later (at bug discovery) by individuals in B and C. –  mljrg Jul 5 '12 at 11:11

The distributions all overlap very much. The Kruskal Wallis test seems to be indicating that the centers of the distributions are nearly the same. The distribution for group is highly skewed do to several very extremely high values. That is what causes the distribution to fail the Shapiro-Wilk test. Th anova F test wrongly interprets group A to have a significantly larger mean because it "ignores" the skewness. The Kruskal Wallis test is giving the appropriate answer while the F test is not.

-
It might be worthy that ANOVA tests are robust against normality assumption. –  hbaghishani Jul 5 '12 at 11:03
That is true of t tests but not the anova F test. To Greg Snow's comment it could be enlightening to look at the bootstrap distribution of the group means (particularly group A) to see how skewed it is. –  Michael Chernick Jul 5 '12 at 11:13
I think your point about "The anova F test wrongly interprets group A to have a significantly larger mean" makes sense in my setting especially after my comment to @GregSnow's answer, where I analyse why Kruskal Wallis seems to be the most appropriate choice here. –  mljrg Jul 5 '12 at 11:19