Non-parametric test if two samples are drawn from the same distribution

Question

I would like to test the hypothesis that two samples are drawn from the same population, without making any assumptions about the distributions of the samples or the population. How should I do this?

From Wikipedia my impression is that the Mann Whitney U test should be suitable, but it does not seem to work for me in practice.

For concreteness I have created a dataset with two samples (a, b) that are large (n=10000) and drawn from two populations that are non-normal (bimodal), are similar (same mean), but are different (standard deviation around the "humps.") I am looking for a test that will recognize that these samples are not from the same population.

Histogram view:

R code:

a <- tibble(group = "a",
            n = c(rnorm(1e4, mean=50, sd=10),
                  rnorm(1e4, mean=100, sd=10)))
b <- tibble(group = "b",
            n = c(rnorm(1e4, mean=50, sd=3),
                  rnorm(1e4, mean=100, sd=3)))
ggplot(rbind(a,b), aes(x=n, fill=group)) +
  geom_histogram(position='dodge', bins=100)

Here is the Mann Whitney test surprisingly (?) failing to reject the null hypothesis that the samples are from the same population:

> wilcox.test(n ~ group, rbind(a,b))

        Wilcoxon rank sum test with continuity correction

data:  n by group
W = 199990000, p-value = 0.9932
alternative hypothesis: true location shift is not equal to 0

Help! How should I update the code to detect the different distributions? (I would especially like a method based on generic randomization/resampling if available.)

EDIT:

Thank you everybody for the answers! I am excitedly learning more about the Kolmogorov–Smirnov which seems very suitable for my purposes.

I understand that the KS test is comparing these ECDFs of the two samples:

Here I can visually see three interesting features. (1) The samples are from different distributions. (2) A is clearly above B at certain points. (3) A is clearly below B at certain other points.

The KS test seems to be able to hypothesis-check each of these features:

> ks.test(a$n, b$n)

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D = 0.1364, p-value < 2.2e-16
alternative hypothesis: two-sided

> ks.test(a$n, b$n, alternative="greater")

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D^+ = 0.1364, p-value < 2.2e-16
alternative hypothesis: the CDF of x lies above that of y

> ks.test(a$n, b$n, alternative="less")

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D^- = 0.1322, p-value < 2.2e-16
alternative hypothesis: the CDF of x lies below that of y

That is really neat! I have a practical interest in each of these features and so it is great that the KS test can check each of them.

Answer 1

The Kolmogorov-Smirnov test is the most common way to do this, but there are also some other options.

The tests are based on the empirical cumulative distribution functions. The basic procedure is:

• Choose a way to measure the distance between the ECDFs. Since ECDFs are functions, the obvious candidates are the $L^p$ norms, which measure distance in function spaces. This distance is our test statistic.
• Figure out the distribution of the test statistic under the null hypothesis that the samples come from the same distribution (luckily people have done this already for the most common distances!)
• Choose a threshold, $\alpha$, for your hypothesis and reject the null if the computed test statistic is in the $\alpha \%$ tails of the distribution from point 2.

For the Kolmogorov-Smirnov test, the test statistic is the maximum distance between the two empirical CDFs (or if you want to be more technical the $L^\infty$ norm). This is super easy to implement in R:

ks.test(a,b)

If the $p$-value is smaller than your chosen threshold, we reject the null hypothesis that the samples are drawn from the same distribution.

Another option is the Cramer-von Mises test, which uses the squared $L^2$ norm as the test statistic and is implemented in the dgof package as cvm.test(). The CVM test is 'better' in the sense that the distance metric takes into account the whole of the two ECDFs, rather than just picking out the largest distance.

EDIT:

Suppose we have samples of size $n$ and $m$, which we want to apply our hypothesis test to.

To turn this into a sampling-type procedure, we can do the following:

1. Generate samples of size $n$ and $m$ from identical distributions. For the KS test (remarkably, IMO) it doesn't even matter if the distribution changes at each iteration as long as $n$ and $m$ stay the same.
2. Calculate your distance metric for the samples. For the KS test, this is just the max. difference between the empirical CDFs.
3. Store the result and go back to step 1.

Eventually you will build up lots of samples from the distribution of the test statistic under the null hypothesis, whose quantiles you can use to conduct your hypothesis test at whichever level of significance you want. For the KS test statistic, this distribution is called the Kolmogorov distribution.

Note that for the KS test, this is just a waste of computational effort because the quantiles are very simply characterised theoretically, but the procedure is generally applicable to any hypothesis test.

Answer 2

The Kolmogorov-Smirnov test is a good start, but building on the same intuition there exist now more powerfull tests.

I would you refere to this, which gives a overview over tests for exactly your purpose. It also has nice ilustrations and explainations whow they are calculated.

The DTS Test is now recomanded. Also see the companion paper at arXiv:2007.01360 or here for details of this test statistic.