I would like to test the hypothesis that two samples are drawn from the same population, without making any assumptions about the distributions of the samples or the population. How should I do this?
From Wikipedia my impression is that the Mann Whitney U test should be suitable, but it does not seem to work for me in practice.
For concreteness I have created a dataset with two samples (a, b) that are large (n=10000) and drawn from two populations that are non-normal (bimodal), are similar (same mean), but are different (standard deviation around the "humps.") I am looking for a test that will recognize that these samples are not from the same population.
a <- tibble(group = "a", n = c(rnorm(1e4, mean=50, sd=10), rnorm(1e4, mean=100, sd=10))) b <- tibble(group = "b", n = c(rnorm(1e4, mean=50, sd=3), rnorm(1e4, mean=100, sd=3))) ggplot(rbind(a,b), aes(x=n, fill=group)) + geom_histogram(position='dodge', bins=100)
Here is the Mann Whitney test surprisingly (?) failing to reject the null hypothesis that the samples are from the same population:
> wilcox.test(n ~ group, rbind(a,b)) Wilcoxon rank sum test with continuity correction data: n by group W = 199990000, p-value = 0.9932 alternative hypothesis: true location shift is not equal to 0
Help! How should I update the code to detect the different distributions? (I would especially like a method based on generic randomization/resampling if available.)
Thank you everybody for the answers! I am excitedly learning more about the Kolmogorov–Smirnov which seems very suitable for my purposes.
I understand that the KS test is comparing these ECDFs of the two samples:
Here I can visually see three interesting features. (1) The samples are from different distributions. (2) A is clearly above B at certain points. (3) A is clearly below B at certain other points.
The KS test seems to be able to hypothesis-check each of these features:
> ks.test(a$n, b$n) Two-sample Kolmogorov-Smirnov test data: a$n and b$n D = 0.1364, p-value < 2.2e-16 alternative hypothesis: two-sided > ks.test(a$n, b$n, alternative="greater") Two-sample Kolmogorov-Smirnov test data: a$n and b$n D^+ = 0.1364, p-value < 2.2e-16 alternative hypothesis: the CDF of x lies above that of y > ks.test(a$n, b$n, alternative="less") Two-sample Kolmogorov-Smirnov test data: a$n and b$n D^- = 0.1322, p-value < 2.2e-16 alternative hypothesis: the CDF of x lies below that of y
That is really neat! I have a practical interest in each of these features and so it is great that the KS test can check each of them.