# Something more powerful than Kruskal-Wallis Test?

I have a sample from a continuous variable under 3 treatments and I want to assess treatment differences in the location of the underlying distributions.

One option is to run a non-parametric test such as the Kruskal-Wallis (kruskal.test in R).

Wondering whether there are different non-parametric tests that I can use.

Simulation example (in R):

n <- 10 # observations per treatment
y <- c(rchisq(n, df=1), rnorm(n), runif(n))
x <- rep(1:3, c(n, n, n))
kruskal.test(y ~ x, data=cbind(x, y))

• If you already feel the distributions are different but don't believe the results of a test that indicates otherwise, it's not clear why you're performing the test. Also if you run tests until one rejects then your testing procedure loses much of its validity. In any event, what are the sample sizes involved? Can you show us plots of the distributions? Aug 30 '16 at 20:58
• You are right the test offers no evidence of a difference. I am just wondering if this result is because the power of the test is extremely low and requires a lot of observations to detect small differences or the test is extremely powerful and I can be safe concluding that there are no differences.
– mrb
Aug 30 '16 at 21:04
• With regards to your request for data or descriptives, I truly appreciate your interest but I am just asking whether there are alternative tests that I can use and not an opinion on my specific dataset
– mrb
Aug 30 '16 at 21:06
• What do the histograms/boxplots look like?
– Jon
Aug 30 '16 at 21:23
• Can you show us an example where you think they're clearly different but the Kruskal-Wallis fails to reject? It would give us a better idea what you're dealing with (or possibly, what you might be doing wrong). There are more powerful tests for particular situations, but not necessarily by very much. It's much more likely there's something important you're not telling us -- possibly several things -- but which we might see if we get to see some data. If you can't show us the data itself, you might scale it. What does the response variable measure -- counts, percentages, 0/1...? Aug 30 '16 at 22:12

I think the issue you're having is both one of sample size and the nature of the alternative hypothesis for the particular test you're using. The Kruskal-Wallis test tries to determine if the distributions are equal, or if one stochastically dominates another. This means that the probability that one quantity is larger than $t$ is greater than that of another distribution for every $t$ (or less precisely the probability that one quantity is "big" is larger than another). The point is that the Kruskal-Wallis test isn't sensitive to any differences whatsoever between the distributions.

If we take your example and plot the empirical distribution functions we will see that the times when the test rejects roughly coincide with the cases where the empirical distribution functions don't overlap. If you're interested not just in stochastic domination but differences in shape as well, you might consider a $k$-sample Kolmogorov-Smirnov test which is discussed in this post: Is there a multiple-sample version or alternative to the Kolmogorov-Smirnov Test?.

library(ggplot2)

n <- 10
y <- c(sort(rchisq(n, df=1)), sort(rnorm(n)), sort(runif(n))
x <- rep(1:3, c(n, n, n))
# calculate empirical distribution functions
f <- rep(1:n / n, 3)
df <- data.frame(x, y, f)
rm(x, y, f, n)

kruskal.test(y ~ x, data=df)

# plot empirical distribution functions
qplot(y, f, data=df, geom="step", colour=as.factor(x))


• My understanding is that Kruskal-Wallis tests the alternative that at least one sample comes from a distribution with a different "location". So, when you say "stochastically dominates" you mean first order, second order stochastic dominance, or both?
– mrb
Aug 31 '16 at 15:06
• The alternative is for first order stochastic dominance. Aug 31 '16 at 15:12

(A little less formal that dsaxton's analysis... but a quick way to judge in this case)

It's not at all clear to me that these are different:

For a rough pairwise comparison, at sample size 10, the uncertainty in the median (since we're looking at boxplots here) is about the size that if the boxes overlap, the two aren't significantly different (though it depends partly on the relative spreads).

[Where does this "at n=10 see if the boxes overlap" idea come from? See the analysis here and then note the neat coincidence that $1.58$ is almost exactly $\sqrt{10}/2$, which means that, at least when the median is toward the middle of the box, the problem reduces to checking for overlapping boxes. I noticed this back in the 80s and it's a rule of thumb that's come in pretty handy. It's not hard to adjust for other sample sizes from there -- e.g. if n is 40, the complete notch interval will be half a box-width]

As we see here, the boxes for groups 1 and 3 overlap almost completely (in that the interval for group 3 is almost entirely contained in that for group 1), and group 2 just overlaps group 3, while groups 1 and 2 just fail to overlap.

Now note that the median for the low group (group 2) is high in its box, not symmetric, while the median for the high group (group 1) is low, so the indication of a difference in location is even less strong there.

So at least looking at the information from the boxplots alone, I see little reason to think there's necessarily anything different here -- far from being obvious, this is pretty equivocal evidence of a difference.

(In fact if you look at notched boxplots, all the pairwise notch intervals overlap substantially.)

So if I were to guess just from the boxplot, I'd think that Kruskal-Wallis test would perhaps be around the border of rejection at the 5% level, but I wouldn't actually expect it to reject. (It might, depending on the specifics of the sample -- it doesn't quite do the same thing as comparing boxplots -- but we really shouldn't be surprised if it doesn't)

So it's not that the Kruskal-Wallis is missing anything here -- I'd say your judgement about what would "appear quite different" (as you put it in the original post) is miscalibrated for this small sample size. The indication of location difference is simply not clear in the data.

If you're interested in more general differences than location differences (such as differences in spread or shape), you might consider other tests than this one... but with an even broader class of alternatives at n=10 you generally won't be able to say much.