Statistical test to tell whether two samples are pulled from the same population?

Question

Let's say I have two samples. If I want to tell whether they are pulled from different populations, I can run a t-test. But let's say I want to test whether the samples are from the same population. How does one do this? That is, how do I calculate the statistical probability that these two samples were pulled from the same population?

Answer 1

The tests that compare distributions are rule-out tests. They start with the null hypothesis that the 2 populations are identical, then try to reject that hypothesis. We can never prove the null to be true, just reject it, so these tests cannot really be used to show that 2 samples come from the same population (or identical populations).

This is because there could be minor differences in the distributions (meaning they are not identical), but so small that tests cannot really find the difference.

Consider 2 distributions, the first is uniform from 0 to 1, the second is a mixture of 2 uniforms, so it is 1 between 0 and 0.999, and also 1 between 9.999 and 10 (0 elsewhere). So clearly these distributions are different (whether the difference is meaningful is another question), but if you take a sample size of 50 from each (total 100) there is over a 90% chance that you will only see values between 0 and 0.999 and be unable to see any real difference.

There are ways to do what is called equivalence testing where you ask if the 2 distributions/populations are equivalent, but you need to define what you consider to be equivalent. It is usually that some measure of difference is within a given range, i.e. the difference in the 2 means is less than 5% of the average of the 2 means, or the KS statistic is below a given cut-off, etc. If you can then calculate a confidence interval for the difference statistic (difference of means could just be the t confidence interval, bootstrapping, simulation, or other methods may be needed for other statistics). If the entire confidence interval falls in the "equivalence region" then we consider the 2 populations/distributions to be "equivalent".

The hard part is figuring out what the equivalence region should be.

Answer 2

http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Assuming your sample values come from continuous distributions, I would suggest the Kolmogorov-Smirnov test. It can be used to test whether two samples come from different distributions (this is how I am interpreting your usage of population) based on their associated empirical distributions.

Directly from Wikipedia:

The null distribution of this statistic is calculated under the null hypothesis that the samples are drawn from the same distribution (in the two-sample case)

The ks.test function in R can be used for this test.

While it is true the kstest does not test for homogeneity, I would argue that if you fail to reject with a large enough sample size (a high powered test), you can claim the differences are not practically significant. You could infer that if differences do exist, they are likely not meaningful (again, assuming large sample size). You cannot conclude they are from the same population as others have correctly stated. All this being said, typically I would just graphically examine the two samples for similarity.

Answer 3

You can use a 'shift function' which checks whether the 2 distributions differ at at each decile. While its technically a test of whether they are from different populations rather than the same, if the distributions don't differ on any of the deciles then you can be reasonably sure they are from the same population, especially if the group sizes are large.

I would also visualize the 2 groups: overlay their distributions and see if they resemble each other, or better yet draw a couple of thousand bootstrap samples from each group and plot those, as this would give you an idea of whether they come from the same population particularly if the population in question isnt normally distributed for you given variable.

Answer 4

I recently had to do something similar (although in my case I needed to know if two distributions were significantly different). Our samples were very large (several hundred to tens of thousands) so KS test said everything was different (even for distributions that appeared nearly identical upon inspection of their histograms).

As AdamO mentions, this is a potential shortcoming of the KS test. Essentially, the KS test examines the maximum difference between the cumulative distributions of each sample...

So if we have 2 distributions, A, and B, one valid question we could ask, is whether the difference between the cdf of A and the cdf of B is large compared with the cdf of a random sample taken from A compared to the cdf for the rest of A or the cdf of a random sample taken from B compared to the cdf for the rest of B.

So the solution I came up with was to take bootstraps of $A$ and $B$ ($A^*$ and $B^*$).

I then constructed an approximation of the null distribution by taking the KS statistic (D statistic, which is the maximum difference in cdf's) of $A^*$ vs $A$ and $B^*$ vs $B$, denoted $KS(A^*,A)$ and $KS(B^*,B)$ respectively. This gives an idea of the range of D values that would be expected if you compared additional samples from either distribution to itself.

Next I got the width of $KS(A^*,A)$ and $KS(B^*,B)$ by subtracting value of the 1st and 99th percentile and dividing by two to yield $sA^*$ and $sB^*$ respectively. Eg. $sA^* = (quantile(KS(A^*,A),.99)-quantile(KS(A^*,A),.01))/2 $

By adding $sA^*$ + $sB^*$ to the the maximum of $<KS(A^*,A)>$ (the expected value of the distribution of $KS(A^*,A)$) and $<KS(B^*,B)>$. I constructed a cutoff threshold for determining whether $KS(A,B)$ was significant. I.e. $Dcut=sA^*+sB^*+max(<KS(A^*,A)>,<KS(B^*,B)>)$

I then computed a confidence interval for the alternative by computing $KS(A^*,B)$ and $KS(B^*,A)$ and used the inner 99% of the resulting D values as a confidence interval for the alternative $KS(A,B)$. I.e. $$CI(KS(A,B)) \sim [quantile(KS(A^*,B) \cup KS(B^*,A),.01),quantile(KS(A^*,B) \cup KS(B^*,A),.99)]$$

Finally, if the confidence interval for the alternative lied above the threshold from the null, $A$ and $B$ were determined to be significantly different (with 99% confidence).

Answer 5

This question came up as the top related result when I was searching for a similar topic, namely, how to compare whether the means of two distributions had the same mean. My question is slightly different because two distributions could be different in ways other than mean (e.g. variance or skew or kurtosis or other "shape measures") but still have nearly equivalent means.

@GregSnow's comment got me on the right track - the term for this is an Equivalence Test.

Another key concept is that you actually cannot compare if two sample distributions are probably the same or not - only if they are close within a bound that you have to pick a priori.

In my case, the Two One-Sided T-Test (TOST) is a recommended procedure and what I used. Once you grok it, it makes sense. It relies on the standard t-test methodology, but uses a trick to make the null hypothesis that the distribution are different (rather than the usual null hypothesis that the distributions are the same).

For my situation, KS was inappropriate because KS DOES test for shape measures, not just means, and I was only interested in means.

Here is the link I used to understand how to implement TOST.

https://www.real-statistics.com/students-t-distribution/equivalence-testing-tost/