Can the Wilcoxon rank sum test give a different result to the Kolmogorov-Smirnov test result?

Question

Let's say I have two data sets (in R, say); $x_1, x_2,..., x_n$ and $y_1, y_2,..., y_n$.

The Wilcoxon rank sum test rejects, indicating that the "X" population distribution differs from that for "Y".

Is it possible that the two sample Kolmogorov-Smirnov test would not indicate that they're different?

Or can we predict that the Wilcoxon would not cause us to reject the null if the Kolmogorov Smirnov test did not?

Answer 1

The crux of @Glen_b's answer (+1) is that these are two different tests that are "designed to pick up... [different and] specific kinds of differences" between the two distributions. So to understand how the results (in terms of whether they are significant or not) can differ between the Wilcoxon rank sum test and the Kolmogorov-Smirnov tests, we need to understand what the tests are designed to detect.

• The Wilcoxon rank sum test tests if:

  the probability of an observation from the population X exceeding an observation from the second population Y equals the probability of an observation from Y exceeding an observation from X: P(X > Y) = P(Y > X) or P(X > Y) + 0.5 · P(X = Y) = 0.5

  That is, it is testing if values of X tend to be larger or smaller than values of Y.

• The Kolmogorov-Smirnov test assesses the largest¹ difference between the two empirical cumulative distribution functions (ECDFs) and compares it to its sampling distribution assuming the distributions are the same.

From here, it is easy to see how there can be datasets where the tests will yield different results.

• The Wilcoxon will be significant while the KS will not when one sample is consistently greater than the other, but not by a large absolute value, and where the distribution shapes are largely the same.

  set.seed(9825)
  g1 = rnorm(10)
  g2 = g1+1.27
  
  wilcox.test(g1, g2)
  #   Wilcoxon rank sum test
  # 
  # data:  g1 and g2
  # W = 22, p-value = 0.03546
  # alternative hypothesis: true location shift is not equal to 0
  ks.test(g1, g2)
  #   Two-sample Kolmogorov-Smirnov test
  # 
  # data:  g1 and g2
  # D = 0.5, p-value = 0.1678
  # alternative hypothesis: two-sided
  

  

• The KS will be significant while the rank sum test will not when the means and medians are the same but the shapes differ markedly.

  set.seed(3806)
  g1 = scale(rnorm(15),       center=TRUE, scale=FALSE)
  g2 = scale(rnorm(15, sd=5), center=TRUE, scale=FALSE)
  
  wilcox.test(g1, g2)
  #   Wilcoxon rank sum test
  # 
  # data:  g1 and g2
  # W = 131, p-value = 0.461
  # alternative hypothesis: true location shift is not equal to 0
  ks.test(g1, g2)
  #   Two-sample Kolmogorov-Smirnov test
  # 
  # data:  g1 and g2
  # D = 0.53333, p-value = 0.02625
  # alternative hypothesis: two-sided
  

  

_{1. More technically the supremum.}

Answer 2

Statistical tests measure differences between samples in different ways and are sensitive to different aspects of the distributional differences.

It's certainly possible for one test to reject and another test (perhaps designed for testing a somewhat different hypothesis) to fail to reject when applied to the same sample at the same significance level. (However, you seem to be misinterpreting what failure to reject implies -- it does not mean that the test shows the population distributions are the same. It only means that the differences in the sample weren't large enough or perhaps were not of the right kind for the test to detect whatever difference there might have been.)

In fact that's what we should hope -- that a test that's designed to pick up more specific kinds of differences should be slightly more sensitive to those kinds of differences that we're most interested in.

In the case of the Kolmogorov-Smirnov test, it's intended to pick up any kind of difference in distributions, but tends to have less power to detect differences in the tails than say Tony Pettitt's two-sample version of the Anderson-Darling (while doing slightly better at differences in the middle). The Wilcoxon-Mann-Whitney by contrast is focused on picking up differences that tend to make one variable typically "larger" than another (alternatives of the form $P(X>Y)>\frac12$ which encompasses location shift alternatives among a host of other possibilities). At the same time, it can't detect a shift in scale that's not accompanied by a difference in location (e.g. a difference in variance of two normals with the same mean). Because its statistic concentrates on a more particular kind of difference in distribution, it does better at seeing when that's the kind of difference that's present.

Here's a pair of samples where the Wilcoxon-Mann-Whitney rejects and the Komogorov-Smirnov does not:

x
 10.110 10.372 10.038 10.159  9.716 10.741 10.637  9.417  9.751 10.014
y
 11.273 11.199 10.575 11.051 10.214 10.448 12.439 10.308 10.225  9.387

Looked at pictorially, the Kolmogorov-Smirnov is looking at the biggest vertical distance between the two ecdfs (which is anywhere just to the left of the second-smallest blue point at 10.214 and to the right of the red point at 10.159); it picks up that there's been a lot of red points with only the one blue one. Meanwhile the Wilcoxon-Mann-Whitney is focused on the tendency to have a lot more red points to the left of blue points than blue points to the left of red points.

As we see here, where the Wilcoxon-Mann-Whitney attains a p-value of 4.3% while the Kolmogorov-Smirnov has a p-value of 5.2%:

> wilcox.test(x,y)

        Wilcoxon rank sum test

data:  x and y
W = 23, p-value = 0.04326
alternative hypothesis: true location shift is not equal to 0

> ks.test(x,y)

        Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.6, p-value = 0.05245
alternative hypothesis: two-sided

In this case I made the data by generating two random (normally distributed, as it happens but this is not important) samples and then shifted one slightly up (just enough to lead the rank sum test to reject). Since it's somewhat more sensitive to that kind of difference than the Kolmogorov-Smirnov two sample test is, there was a good chance that the Kolmogorov-Smirnov test might not pick up the difference at that significance level (and that's what happened).