Kruskal-Wallis vs Jonckheere-Terpstra Test

If we're doing a Jonckheere-Terpstra Test and a Kruskal-Wallis test at the same significance level, which one is more likely to reject the null hypothesis when it is actually true?

From my understanding, the Jonckheere-Terpstra test tests the ordering of medians versus the Kruskal-Wallis test, that tests the equality of medians. So the former can give a significant result if the medians are actually equal, which Kruskal-Wallis will not. Hence the former is more likely to reject the null than the latter. Does this make sense? An explanation will be helpful.

You are correct that the Kruskal-Wallis test ignores ordering—making it akin to a one-way ANOVA performed on ranks. However, you are a bit off with the null hypothesis of the Kruskal-Wallis which takes the form:

$$H_{0}: P\left(X_{i} > X_{j}\right) = 0.5\\H_{A}: P\left(X_{i} > X_{j}\right) \ne 0.5$$

for all groups $i\ne j$. In plain language the null is that the probability of a randomly selected observation in one group being greater than a randomly selected observation in another group is one half.

The Kruskal-Wallis test only becomes an omnibus test for difference of unordered medians with two additional assumptions:

1. The distributions of each group all have the same shape—whatever that shape actually is.
2. The distributions of each group all have the same variance.

These two strict assumptions make Kruskal-Wallis a test for location shift (more general than a test for mean difference, or a test for median difference).

Given that the Kruskal-Wallis test ignores ordering, ceteris paribus the Jonckheere-Terpstra test will be more powerful. Indeed, Jonckheere noted that the test in balanced data was more powerful than a one-way ANOVA, which also ignores ordering.

You write "So the former can give a significant result if the medians are actually equal, which Kruskal-Wallis will not," but this is untrue: the Kruskal-Wallis test certainly can reject the null hypothesis even when it is true. In fact, when you choose $\alpha$ as a rejection criterion for this test you are selecting your preference for making exactly this kind of Type I error. Jonckheere gives a caution about using the Jonckheere-Terpstra test when the sample size in one group is large, but very small in other groups, as this will make the limiting distribution of the approximate test statistic non-normal, making inferences unreliable.

• Thanks! The Jonckheere-Terpstra test will be more powerful - but does that also mean that it is more likely to reject the null when it is actually true? – Abominable Snowman Mar 24 '18 at 17:04
• @AbominableSnowman Ah... I misunderstood your question... you are concerned about Type I errors. See my edit? – Alexis Mar 24 '18 at 17:16
• Ok. So we can't say for certain whether one will always have a higher type 1 error rate than the other, and it depends on the data at hand. – Abominable Snowman Mar 24 '18 at 17:17
• @AbominableSnowman Specifically it depends on the balance of your design (equality of sample sizes in groups), and sample sizes for the asymptotic tests. With small sample sizes you would just use the exact p-values for either test, and then your Type I error rate is just $\alpha$. – Alexis Mar 24 '18 at 17:35