I am currently working with data in which I have one continuous variable (speed) and a categorical variable (3 different populations, with n ranging from 13 to 30 for each group). I have determined that my data are not normal using a Shapiro-Wilk test, but are similar in variance (assessed using Levene's test). My research indicates that ANOVAs are robust against violations of normality, and I prefer the ANOVA over Kruskal-Wallis since K-W is a rank test.

Out of curiosity I ran both the ANOVA and Kruskal-Wallis test and got a lower p-value from the Kruskal-Wallis. I thought this might be odd since my research indicates that if violation of normality is affecting results of an ANOVA it would do so as an increased chance of Type I errors. Does this result seem odd? Or could it be interpreted as an indication that using the ANOVA would be acceptable?

Edit: The data are listed below. I've also included a link to a google sheet in case that's easier: link

  population   speed  
1          sf 15.4430  
2          sf 12.6530  
3          sf 13.1080  
4          sf 14.0120  
5          sf 13.4710  
6          sf 21.3610  
7          sf 15.2970  
8          sf 13.5380  
9          sf 13.5800  
10         sf 14.1760  
11         sf 14.2330  
12         sf 13.4240  
13         sf 15.0790  
14         pa  8.7742  
15         pa  7.6776  
16         pa 10.6600  
17         pa  9.9945  
18         pa 10.4490  
19         pa 11.3420  
20         pa 13.0910  
21         pa 10.7780  
22         pa 15.7640  
23         pa 12.3630  
24         pa 11.4780  
25         pa 11.5700  
26         pa  7.4900  
27         pa 13.0600  
28         pa 13.4490  
29         f2 14.1640  
30         f2 14.6710  
31         f2 13.7380  
32         f2 14.9100  
33         f2 13.9390  
34         f2 12.1680  
35         f2 10.7940  
36         f2 11.0920  
37         f2 13.6060  
38         f2 13.0190  
39         f2 13.9010  
40         f2 11.4980  
41         f2 14.6970  
42         f2 14.3190  
43         f2 13.0460  
44         f2 12.8570  
45         f2 12.3970  
46         f2 15.2730  
47         f2 12.8950  
48         f2  7.7650  
49         f2 13.1850  
50         f2  7.9470  
51         f2 11.6020  
52         f2 13.3950  
53         f2 12.4550  
54         f2 14.0910  
55         f2 14.8370  
56         f2 14.5250  
57         f2 12.1720  
58         f2 12.7710  
59         f2 15.2850  
60         f2 12.9380  
  • 1
    $\begingroup$ What precisely did you feed to Shapiro-Wilk? The normality assumption is about errors, not the marginal distribution of the response. Otherwise there is no way that K-W uses precisely the same information, e.g. outliers will be suppressed on conversion to ranks, and there are many ways in which results might not be consistent. You have somewhere between 39 and 90 data points, as I understand it, so why not post them here to make matters concrete? $\endgroup$
    – Nick Cox
    Jan 10, 2020 at 16:30
  • 1
    $\begingroup$ I'd expect speeds to be at least slightly skewed. Often their reciprocals, in effect times, are more nearly normal and better behaved. But no promises possible without seeing the data. $\endgroup$
    – Nick Cox
    Jan 10, 2020 at 16:32
  • $\begingroup$ Thanks for the reply and the edit. I've added the data. You're correct that the data appear skewed $\endgroup$
    – Alex
    Jan 10, 2020 at 17:29
  • $\begingroup$ Both p-values are extremely small (they're telling you the same thing about the data - that the locations are not all the same). It's unwise to try to put much interpretation on a small difference in p-vales in any case; minor sample fluctuations of no consequence (a little noise) can do more than that. $\endgroup$
    – Glen_b
    Jan 11, 2020 at 3:46

3 Answers 3


Thanks for posting the data. Some exploration and analysis lead to the following suggestions.

  1. The data show some moderate outliers. There is always a danger of overinterpreting small samples, meaning that the gaps might well be filled in by other values in a larger sample.

  2. The idea of a reciprocal transformation, mapping speed to some measure of time spent, makes some details better and some worse, and so is not recommended especially.

  3. The data show some non-normality but I would be pretty happy proceeding to an analysis of variance, with a little circumspection. This plot shows all the data as points, a series of quantiles, conventional median-quartile boxes, and means using horizontal lines. Real data is not often very much better than this. This so-called quantile-box plot emphasises that half the data are outside the boxes as well as half being inside.

enter image description here

  1. If this dataset were mine to analyse, with nothing else known than is stated here, I would want any details that were available about individual measurements; and would probably consider gamma distributions as an alternative reference.
  • $\begingroup$ Thanks so much. I have an additional question if that's alright. I've taken multiple stats classes at uni and none have been helpful (as evidenced by my inability to choose between an ANOVA and kuskal wallis). Are you aware of a resource that is particularly useful for learning more advanced stats than the basic nonsense taught in non-math major classes? thanks again. $\endgroup$
    – Alex
    Jan 10, 2020 at 19:32
  • $\begingroup$ Tell me your substantive field and I might be able to make suggestions. $\endgroup$
    – Nick Cox
    Jan 10, 2020 at 19:37
  • $\begingroup$ Neuroscience, focusing on behavior. Thanks! $\endgroup$
    – Alex
    Jan 13, 2020 at 1:12
  • 1
    $\begingroup$ springer.com/gp/book/9781461496014 $\endgroup$
    – Nick Cox
    Jan 13, 2020 at 8:35

One can only hope that when you applied the Shapiro-Wilk test you did it to the residuals, rather than to the untransformed response, because the latter approach is wrong. Indeed it's often quoted, as you have here, that the assumption is that the data are normal and that's just unclear nonsense. It's the conditional $Y$, the residuals, that have to be normal for exact, finite sample inference to be correct. Even so....

We say ANOVA is robust to the so-called "normality assumption". This is because in moderately large samples, the sampling distribution of the mean is close to normal, and the finite sample vs. asymptotic test (think F vs. Chi-square stat) are nearly identical in inference. For 3 different populations of an $n$ of 13 to 30 for each group, the properties of the central limit theorem certainly make a possibly huge impact. In fact, you can be more assured of that than you can of any reliability whatsoever in visually inspecting, or testing, normality.

Further, when the assumptions of a test are violated, it's not guaranteed that the resulting effect is that the test is of lower power. That would be a desirable property to be certain. But alas, there's no predicting what will happen. In your case, where the normal test is more significant, one of two things is true: either the data are close enough to normal that the ANOVA has much higher power than Kruskal-Wallis; or the distribution is non-normal (or heteroscedastic) in such a way that a highly variable estimate of the mean has a spuriously significant finding.

Lastly, it doesn't make sense to compare the results of two tests when you are uncertain whether assumptions hold for one test but not the other. Finding discordant findings doesn't "enrich" an analysis in any sense, and all too often the result of such fishing expeditions is that the analyst reports "the more favorable test"'s result, without any correction for multiple comparisons and consequently doubles the false positive error rate.

  • $\begingroup$ I used graphpad prism to run the shapiro wilk test by feeding it the measurements (not residuals) I'm currently trying to figure out whether it calculates the residuals and runs the test on those or runs it on the given data. Same for R. Do you happen to know whether either R (shapiro.test) or prism calculate residuals themselves? thanks! $\endgroup$
    – Alex
    Jan 10, 2020 at 17:43
  • $\begingroup$ What exactly did you stick into R's shapiro.test? $\endgroup$
    – Dave
    Jan 10, 2020 at 17:45
  • $\begingroup$ shapiro.test(cleandata$speed) where cleandata is the data frame that's in the original post. Thanks! $\endgroup$
    – Alex
    Jan 10, 2020 at 17:56
  • 4
    $\begingroup$ That is not what you should have done. As others noted, it's the residuals that need to be tested. $\endgroup$
    – Peter Flom
    Jan 10, 2020 at 18:51
  • $\begingroup$ +1. I especially like your last paragraph. In a similar fashion, it is not uncommon to see people run $t$-tests before doing a regression, or compare their GLM to an ordinary linear regression where only one is of interest/appropriate. $\endgroup$ Jan 11, 2020 at 9:47

You asked:

"Does this result seem odd?"

Not particularly. Even if the assumptions of ANOVA are met, KW and ANOVA ask different questions. It is not surprising that they give different answers. (KW asks about "stochastic dominance" and uses ranks; ANOVA compares means). And, while the power of ANOVA is greater when the assumptions are met, when they are not met, anything can happen.

Or could it be interpreted as an indication that using the ANOVA would be acceptable?

No. Whether ANOVA is acceptable depends on whether the assumptions are met.

I looked at your data in SAS using this code (you could do something similar in R

proc glm data = junk plots = all;
 class grp;
 model value = grp;

The quantile normal plot of the residuals (which is a better method than any test) shows fairly marked differences from normality. Not only is there heteroscedasticy, there are also outliers.

Rather than KW, however, I suggest quantile regression (which also makes no assumptions about the residuals, but is, I think, a closer match to ANOVA) or robust regression (which ameliorates outliers and is quite close to ANOVA in spirit.

Quantile regression on the median yields:

$14.01 - 0.83*F2 - 2.67*pa$

with the coef. for pa being significant (and sf as the reference category).

Robustreg with MM estimation yields

$13.94 -0.61*F2 - 2.77*pa$ again the coef for pa is significant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.