What test do I use to check if two samples came from different population?

Question

I'm aware that t-test checks if two sample data sets have the same difference in means with confidence. But testing difference in means is not equivalent to testing if the two distributions came from the same population.

I'm also aware that Wilcoxon-test and KS-test are non-parametric methods to compare two distributions. But I'm confused if they truly compare "if two distributions came from the sample population."

What other hypothesis tests are available to test if two distributions came from the same population, both parametric and non-parametric?

Answer 1

Your question is too broad for a thorough discussion here. Of the tests you mentioned, the Kolmogorov-Smirnov test is the only one generally regarded as a straightforward test of the difference between two distributions.

Consider Sample 1 of size 100 from $\mathsf{Norm}(\mu=100,\sigma=10)$ and Sample 2 of size 110 from $\mathsf{Norm}(98, 12).$ These two distributions are sufficiently similar that a reasonably large sample is necessary if we are to have any chance of distinguishing samples from them as coming from different distributions. So I chose sample sizes around 100.

set.seed(2019)
x = rnorm(100, 100, 10);  y = rnorm(110, 98, 12)

summary(x); sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  77.37   93.53   97.68   99.27  105.68  126.36 
[1] 9.054535
summary(y); sd(y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  66.01   88.31   95.94   96.01  104.92  124.35 
[1] 11.84239

Summaries show that sample means and standard deviations give reasonably good estimates of the corresponding population parameters.

A Kolmogorov-Smirnov test (ks.test in R) is (barely) able to detect at the 5% level that the samples came from different distributions:

ks.test(x,y)

        Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.18909, p-value = 0.04723
alternative hypothesis: two-sided

Moreover, the K-S test is moderately intolerant of ties. If we round the values in the two samples to integers, thus inducing ties, then the (approximate) P-value increases. (Subsequently, we return to unrounded data.)

X = round(x);  Y = round(y);  ks.test(X,Y)

        Two-sample Kolmogorov-Smirnov test

data:  X and Y
D = 0.17909, p-value = 0.06946
alternative hypothesis: two-sided

Warning message:
In ks.test(X, Y) : p-value will be approximate 
    in the presence of ties

Below we show kernel density estimators (roughly, 'smoothed histograms') of the two samples in the left panel. They show that Sample 1 (with mean 99.27) has generally larger values than does Sample 2 (mean 96.01).

The right panel shows empirical cumulative distribution functions of the two samples. ECDFs, which increase by $1/n$ at each sorted value of the sample, are estimates of the CDFs of the distributions from which we sampled. The K-S test statistic $D = 0.179$ is the largest vertical discrepancy between the two ECDFs. [For further details of the K-S test perhaps you can start with Wikipedia] under the two-sample heading.]

par(mfrow=c(1,2))
 hdr1 = "KDEs of Sample 1 and (red) Sample 2"
 plot(density(x), xlab="",main=hdr1)
  lines(density(y), col="red")
 hdr2 ="ECDFs of Sample 1 and (red) Sample 2"
 plot(ecdf(x), main=hdr2)
  lines(ecdf(y), col="red", pch="+")
par(mfrow=c(1,1))

Finally, the two-sample Wilcoxon (rank sum) test [also called the Mann-Whitney test] is often said to test whether the parent distribution of one sample 'dominates' (is generally larger than) the distribution of another. The one-sided Wicoxon SR test

wilcox.test(x,y, alt="gr")

      Wilcoxon rank sum test with continuity correction

data:  x and y
W = 6331, p-value = 0.02949
alternative hypothesis: 
   true location shift is greater than 0

The P-value is roughly the same for the rounded data, but with a warning message about ties.

Perhaps others on this site will want to explain additional tests relevant to your Question.

Answer 2

A non-parametric test for whether the samples came from the same population is the permutation test. Let's say you have two samples $X$ and $Y$ from distributions with cumulative distribution functions $F_X$ and $F_Y$ respectively. We want to reject the null

$H_0: F_X = F_Y$

The procedure is as follows:

1. Calculate the difference in means in the actual sample. Call it $t_{obs}$.

2. Now permute the data (as in, randomly assign each data point to one of the groups). Recalculate the test statistic (ie. difference in means of the two groups). Let's call it T.

3. Repeat step 2 a large number of times (say B). Then, the p-value is

   $\dfrac{1}{B} \sum_{i=1}^B I(T_i>t_{obs})$

More details can be found here

Answer 3

Kolmogorov-Smirnov test can be used to check whether a sample is from a given distribution (one sample KS test) or to test whether two samples are having same underlying distribution (two sample KS test). So, you can use 2-sample KS test. Another option is Mann–Whitney U test. I cannot tell which one you need to use unless you provide more details about your problem. If I want to shortly explain, KS-test is based on differences in the empirical CDF's of the samples and is sensative to shapes, spreads or medians. On the other hand, Mann–Whitney U test works based on differences between the mean ranks of the two groups.

Answer 4

The Baumgartner-Weiss-Schindler test is "a non-parametric hypothesis test for the null of equal probability distributions of two samples" that is "more powerful than ... other tests under certain alternatives." In R:

require(BWStest)
set.seed(2019)
x = rnorm(100, 100, 10);  y = rnorm(110, 98, 12)
bws_test(x,y)


        two-sample BWS test

data:  x vs. y
B = 3.5, p-value = 0.02
alternative hypothesis: true difference in survival functions is not equal to 0