19
$\begingroup$

I would like to test the hypothesis that two samples are drawn from the same population, without making any assumptions about the distributions of the samples or the population. How should I do this?

From Wikipedia my impression is that the Mann Whitney U test should be suitable, but it does not seem to work for me in practice.

For concreteness I have created a dataset with two samples (a, b) that are large (n=10000) and drawn from two populations that are non-normal (bimodal), are similar (same mean), but are different (standard deviation around the "humps.") I am looking for a test that will recognize that these samples are not from the same population.

Histogram view:

bimodal samples

R code:

a <- tibble(group = "a",
            n = c(rnorm(1e4, mean=50, sd=10),
                  rnorm(1e4, mean=100, sd=10)))
b <- tibble(group = "b",
            n = c(rnorm(1e4, mean=50, sd=3),
                  rnorm(1e4, mean=100, sd=3)))
ggplot(rbind(a,b), aes(x=n, fill=group)) +
  geom_histogram(position='dodge', bins=100)

Here is the Mann Whitney test surprisingly (?) failing to reject the null hypothesis that the samples are from the same population:

> wilcox.test(n ~ group, rbind(a,b))

        Wilcoxon rank sum test with continuity correction

data:  n by group
W = 199990000, p-value = 0.9932
alternative hypothesis: true location shift is not equal to 0

Help! How should I update the code to detect the different distributions? (I would especially like a method based on generic randomization/resampling if available.)

EDIT:

Thank you everybody for the answers! I am excitedly learning more about the Kolmogorov–Smirnov which seems very suitable for my purposes.

I understand that the KS test is comparing these ECDFs of the two samples:

ECDFs

Here I can visually see three interesting features. (1) The samples are from different distributions. (2) A is clearly above B at certain points. (3) A is clearly below B at certain other points.

The KS test seems to be able to hypothesis-check each of these features:

> ks.test(a$n, b$n)

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D = 0.1364, p-value < 2.2e-16
alternative hypothesis: two-sided

> ks.test(a$n, b$n, alternative="greater")

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D^+ = 0.1364, p-value < 2.2e-16
alternative hypothesis: the CDF of x lies above that of y

> ks.test(a$n, b$n, alternative="less")

        Two-sample Kolmogorov-Smirnov test

data:  a$n and b$n
D^- = 0.1322, p-value < 2.2e-16
alternative hypothesis: the CDF of x lies below that of y

That is really neat! I have a practical interest in each of these features and so it is great that the KS test can check each of them.

$\endgroup$
  • $\begingroup$ It is not too surprising that MW does not reject. For a one-sided test it tests whether Pr(a>b) < 0.05 where a and b are randomly chosen members of your populations. $\endgroup$ – mdewey Jul 2 '17 at 14:00
  • 1
    $\begingroup$ The hypothesis for Mann-Whitney is sometimes said to concern "location" of the two groups, or something along the lines of systematic stochastic difference. In the case of your data, both groups are symmetrically distributed around 75, so M-W should definitely not find a difference. $\endgroup$ – Sal Mangiafico Jul 2 '17 at 14:54
  • 4
    $\begingroup$ This is a good example of the confusion we sow when we are not clear about the hypothesis for a test. Unfortunately, people are taught to use a t-test to compare two groups, without really thinking that this test compares two means, whereas there's a median test to compare two medians, Mann-Whitney that compares something else, quantile regression to compare other percentiles, tests to compare variances, Kolmogorov-Smirnov to compare the distributions, and so on... We sometimes just say we want to compare two "populations" without being clear which hypothesis we really want to test. $\endgroup$ – Sal Mangiafico Jul 2 '17 at 15:05
  • $\begingroup$ On reflection it seems like the Wikipedia page for the M-W test states the hypothesis very clearly and it was a misunderstanding (unfounded leap) on my part to think that this hypothesis also implies that the samples come from the same distribution. Indeed, the problem becomes obvious when comparing two different distributions that are symmetric around the same central point. $\endgroup$ – Luke Gorrie Jul 3 '17 at 7:17
17
$\begingroup$

The Kolmogorov-Smirnov test is the most common way to do this, but there are also some other options.

The tests are based on the empirical cumulative distribution functions. The basic procedure is:

  • Choose a way to measure the distance between the ECDFs. Since ECDFs are functions, the obvious candidates are the $L^p$ norms, which measure distance in function spaces. This distance is our test statistic.
  • Figure out the distribution of the test statistic under the null hypothesis that the samples come from the same distribution (luckily people have done this already for the most common distances!)
  • Choose a threshold, $\alpha$, for your hypothesis and reject the null if the computed test statistic is in the $\alpha \%$ tails of the distribution from point 2.

For the Kolmogorov-Smirnov test, the test statistic is the maximum distance between the two empirical CDFs (or if you want to be more technical the $L^\infty$ norm). This is super easy to implement in R:

ks.test(a,b)

If the $p$-value is smaller than your chosen threshold, we reject the null hypothesis that the samples are drawn from the same distribution.

Another option is the Cramer-von Mises test, which uses the squared $L^2$ norm as the test statistic and is implemented in the dgof package as cvm.test(). The CVM test is 'better' in the sense that the distance metric takes into account the whole of the two ECDFs, rather than just picking out the largest distance.

EDIT:

Suppose we have samples of size $n$ and $m$, which we want to apply our hypothesis test to.

To turn this into a sampling-type procedure, we can do the following:

  1. Generate samples of size $n$ and $m$ from identical distributions. For the KS test (remarkably, IMO) it doesn't even matter if the distribution changes at each iteration as long as $n$ and $m$ stay the same.
  2. Calculate your distance metric for the samples. For the KS test, this is just the max. difference between the empirical CDFs.
  3. Store the result and go back to step 1.

Eventually you will build up lots of samples from the distribution of the test statistic under the null hypothesis, whose quantiles you can use to conduct your hypothesis test at whichever level of significance you want. For the KS test statistic, this distribution is called the Kolmogorov distribution.

Note that for the KS test, this is just a waste of computational effort because the quantiles are very simply characterised theoretically, but the procedure is generally applicable to any hypothesis test.

$\endgroup$
  • $\begingroup$ Thank you! The Kolmogorov-Smirnov test does indeed reject the null hypothesis that these samples are from the same population. And intuitively it makes sense to compare the ECDFs because that is more-or-less what I am doing visually with the histogram. Question: Suppose that I needed to implement this test from scratch without any tools like R. Is there a simple method that would suffice? (Perhaps based on bootstrapping?) I ask because my background is computer programming and I find simulation-based methods much easier to really understand. $\endgroup$ – Luke Gorrie Jul 2 '17 at 12:13
  • $\begingroup$ You should look into randomization or permutations. I prefer these for non-normal tests. They also meet ur criteria of being simulation rather than statistics $\endgroup$ – RTbecard Jul 2 '17 at 13:16
  • 2
    $\begingroup$ @JamesAdamCampbell could you expand on that a bit in a different answer? $\endgroup$ – Will Jul 2 '17 at 13:41
  • 1
    $\begingroup$ @LukeGorrie, how to implement the Kolmogorov-Smirnov test is a separate question -- and one that might be better posted elsewhere (e.g., on Stack Overflow). The basic recipe is: (a) understand how the test works, (b) implement that. In this case it seems particularly straightforward: 1. compute the ECDFs (easy, by just taking prefix sums), 2. compute the $L_\infty$ norm (easy, by just taking the max difference). It looks pretty straightforward. $\endgroup$ – D.W. Jul 2 '17 at 15:47
  • 1
    $\begingroup$ I don't see any problem with that. I'd be interested to see the results if you try some of this stuff! Would be cool to see whether the CI approach and straight up KS test always give you the same answer. I suspect they do :) $\endgroup$ – Will Jul 2 '17 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.