using this code

pvalue = rep(NA, 100000)
n = c(352, 198, 170) 
group = rep(1:3, n)
for (i in 1:100000){
x=c(rnbinom(352, size=0.9563, mu=2.27), rnbinom(198, size=1.0468, mu=2.27), rnbinom(170, size=1.3264, mu=2.27))
 kruskal = kruskal.test(x ~ group)
pvalue[i] = kruskal$p.value}
result = length(pvalue[pvalue<=0.05])/100000    

I tried to examine the risk of type 1 error when comparing three neg. binomial distributions with same mu and different variances using the Kruskal Wallis Test. To my astonishment, I found out, that 8.3% of the 100,000 conducted tests had a p-value lower than 5%.

I still remember from my statistics lessons, that non-parametric tests like the Kruskal Wallis are said to be more conservative and, if I remember correctly, less likely to lead to a type 1 error compared to parametric tests, when applied to data which violate certain assumptions like homogeneous variances or normal distribution.

However, in this example, Kruskal Wallis apparently led to a higher type 1 error. I would be thankful, if someone could confirm to me, if the Kruskal Wallis Test could possibly be that anti-conservative or if my code is maybe wrong? However, the result seems logical, as different variances in the samples could lead to the conclusion, that they stem from different populations... BTW, I know there are better methods to examine the distributions at hand, I just tried out the Kruskal Wallis Test "for fun". Thank you very much in advance!

Doing the same with ANOVA instead of Kruskal Wallis, I get a lower result than 5%:

> pvalue = rep(NA, 100000)
> n = c(352, 198, 170)    
> group = rep(1:3, n)  
> for (i in 1:100000){  
+   x=c(rnbinom(352, size=0.9563, mu=2.27), rnbinom(198, size=1.0468, mu=2.27), rnbinom(170, size=1.3264, mu=2.27))  
+   result = aov(x ~ factor(group))  
+   pvalue[i] = anova(result)[1,5]}  
> robust = length(pvalue[pvalue<=0.05])/100000  
> robust 
[1] 0.0438    

Probably there is something wrong with the ANOVA-code, but I can't figure out what. I would appreciate any help!

  • $\begingroup$ Much better question but always test p < 0.05 not <=. $\endgroup$ – John Sep 7 '13 at 21:26
  • $\begingroup$ Why do you claim that $H_0$ is fulfilled for these negative binomial distributions ? They do not have the same median (I don't know exactly what is the location parameter in the $H_0$ hypothesis of KW). $\endgroup$ – Stéphane Laurent Nov 10 '13 at 16:46

The nonparametric test is supposed to be more conservative than the ANOVA in cases of violation but that doesn't mean it is in all cases. I've looked at a few types of distributions, varying N's and variability of distributions and I can get this to slide around a lot of ways. For example, it's well known that you can easily make an ANOVA very conservative with unequal N's and variances among conditions. That doesn't impact the Kruskal-Wallis test the same way.

In short, you can easily create cases where the conservativeness of a nonparametric test doesn't hold and where alpha is violated. I'm not well versed in the research on this specific nonparametric test but one of the advantages of them is that at least then your conclusions are always more conservative. The conclusions drawn from parametric tests relate to parameters in the population whereas the conclusions from nonparametric tests relate to this particular set of data and the likelihood of your finding.

Fortunately, you're skilled enough to do simulations and test what is an appropriate statistic to use in your case. :)

  • $\begingroup$ Many thanks! I did perform the same thing with ANOVA as well with result = aov(x ~ factor(group)) pvalue[i] = anova(result)[1,5]} length(pvalue[pvalue<=0.05])/100000 (the part above is the same) and got [1] 0.0461 as an answer. Thus, there seems to be something wrong with my ANOVA code, I'll check. $\endgroup$ – Renoir Pulitz Sep 7 '13 at 19:58
  • $\begingroup$ See my edits. To tell the truth, I knew when writing the prior answer that it was just fortunate things came out that way...even if they really had come out that way. You could have easily been right for a specific case. Regardless, I'm glad that the original answer helped improve the question. $\endgroup$ – John Sep 7 '13 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.