# Statistical test to tell whether two samples are pulled from the same population?

Let's say I have two samples. If I want to tell whether they are pulled from different populations, I can run a t-test. But let's say I want to test whether the samples are from the same population. How does one do this? That is, how do I calculate the statistical probability that these two samples were pulled from the same population?

• Please explain--as quantitatively as possible--what you mean by "same." It would help to clarify what you mean by "condition," too.
– whuber
Jan 23, 2014 at 20:52
• A test like a two-sample Kolmogorov-Smirnov (not the only possibility; with the usual assumptions, the t-test is testing the same thing, as you note) can test whether the population distributions are different (but failure to reject doesn't mean they're actually the same). However, no test can tell you whether two distributions that are not too dissimilar are actually from the same population, rather than two different populations with similar distributions. That would have to come either from assumptions or other investigation. ...ctd Jan 23, 2014 at 22:56
• ctd ... Similarly, tests can't even tell you that the distributions are identical, since they can differ in trivial ways. You may want to search on 'equivalence test' or 'equivalence testing', on which you should get quite a few hits here, or on google. Jan 23, 2014 at 23:00

The tests that compare distributions are rule-out tests. They start with the null hypothesis that the 2 populations are identical, then try to reject that hypothesis. We can never prove the null to be true, just reject it, so these tests cannot really be used to show that 2 samples come from the same population (or identical populations).

This is because there could be minor differences in the distributions (meaning they are not identical), but so small that tests cannot really find the difference.

Consider 2 distributions, the first is uniform from 0 to 1, the second is a mixture of 2 uniforms, so it is 1 between 0 and 0.999, and also 1 between 9.999 and 10 (0 elsewhere). So clearly these distributions are different (whether the difference is meaningful is another question), but if you take a sample size of 50 from each (total 100) there is over a 90% chance that you will only see values between 0 and 0.999 and be unable to see any real difference.

There are ways to do what is called equivalence testing where you ask if the 2 distributions/populations are equivalent, but you need to define what you consider to be equivalent. It is usually that some measure of difference is within a given range, i.e. the difference in the 2 means is less than 5% of the average of the 2 means, or the KS statistic is below a given cut-off, etc. If you can then calculate a confidence interval for the difference statistic (difference of means could just be the t confidence interval, bootstrapping, simulation, or other methods may be needed for other statistics). If the entire confidence interval falls in the "equivalence region" then we consider the 2 populations/distributions to be "equivalent".

The hard part is figuring out what the equivalence region should be.

• A null hypothesis test can never provide evidence for the null hypothesis, true. Model selection, either Bayesian or based on some "criterion" (AIC, BIC) could however indicate that a null model (identical distributions) is a better description of the data than an alternative model (different distribution). All that under a bunch of assumptions of course. Feb 4, 2015 at 2:23

http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Assuming your sample values come from continuous distributions, I would suggest the Kolmogorov-Smirnov test. It can be used to test whether two samples come from different distributions (this is how I am interpreting your usage of population) based on their associated empirical distributions.

Directly from Wikipedia:

The null distribution of this statistic is calculated under the null hypothesis that the samples are drawn from the same distribution (in the two-sample case)

The ks.test function in R can be used for this test.

While it is true the kstest does not test for homogeneity, I would argue that if you fail to reject with a large enough sample size (a high powered test), you can claim the differences are not practically significant. You could infer that if differences do exist, they are likely not meaningful (again, assuming large sample size). You cannot conclude they are from the same population as others have correctly stated. All this being said, typically I would just graphically examine the two samples for similarity.

• I doubt that KS test can be used to show distributional equivalence. Jan 23, 2014 at 22:35
• @MichaelMayer that's exactly right. OP is interested in a test of homogeneity... which has many underlying methodological issues. KS for heterogeneity also has its issues: practically, it will reject in large samples regardless of whether populations are practically identical in all aspects. It just goes to show that testing, and consequently p-values, are better conceived of as measures of sample size than statistical significance. Jan 24, 2014 at 3:33
• @AdamO Yes, but if you have large samples and you fail to reject, I would be confident that the populations are practically identical. As far as I know, there is no theory to back this up, but from experience, knowing that the KS for heterogeneity can detect minute differences with large sample size can allow you to use a failed large sample test as a de facto statement of practically identical populations. Does my response answer the question "calculate the statistical probability that these two samples were pulled from the same population"? Certainly not. Apr 29, 2014 at 20:52
• What can I do if my points are two-dimensional? That is, I have two samples of two-dimensional points, and I want to know if they come from distinct distributions. Oct 28, 2017 at 19:12
• @qwr that is not true. There is a two-sample KS test, defined on wikipedia. Apr 5, 2019 at 16:05

You can use a 'shift function' which checks whether the 2 distributions differ at at each decile. While its technically a test of whether they are from different populations rather than the same, if the distributions don't differ on any of the deciles then you can be reasonably sure they are from the same population, especially if the group sizes are large.

I would also visualize the 2 groups: overlay their distributions and see if they resemble each other, or better yet draw a couple of thousand bootstrap samples from each group and plot those, as this would give you an idea of whether they come from the same population particularly if the population in question isnt normally distributed for you given variable.

I recently had to do something similar (although in my case I needed to know if two distributions were significantly different). Our samples were very large (several hundred to tens of thousands) so KS test said everything was different (even for distributions that appeared nearly identical upon inspection of their histograms).

As AdamO mentions, this is a potential shortcoming of the KS test. Essentially, the KS test examines the maximum difference between the cumulative distributions of each sample...

So if we have 2 distributions, A, and B, one valid question we could ask, is whether the difference between the cdf of A and the cdf of B is large compared with the cdf of a random sample taken from A compared to the cdf for the rest of A or the cdf of a random sample taken from B compared to the cdf for the rest of B.

So the solution I came up with was to take bootstraps of $$A$$ and $$B$$ ($$A^*$$ and $$B^*$$).

I then constructed an approximation of the null distribution by taking the KS statistic (D statistic, which is the maximum difference in cdf's) of $$A^*$$ vs $$A$$ and $$B^*$$ vs $$B$$, denoted $$KS(A^*,A)$$ and $$KS(B^*,B)$$ respectively. This gives an idea of the range of D values that would be expected if you compared additional samples from either distribution to itself.

Next I got the width of $$KS(A^*,A)$$ and $$KS(B^*,B)$$ by subtracting value of the 1st and 99th percentile and dividing by two to yield $$sA^*$$ and $$sB^*$$ respectively. Eg. $$sA^* = (quantile(KS(A^*,A),.99)-quantile(KS(A^*,A),.01))/2$$

By adding $$sA^*$$ + $$sB^*$$ to the the maximum of $$$$ (the expected value of the distribution of $$KS(A^*,A)$$) and $$$$. I constructed a cutoff threshold for determining whether $$KS(A,B)$$ was significant. I.e. $$Dcut=sA^*+sB^*+max(,)$$

I then computed a confidence interval for the alternative by computing $$KS(A^*,B)$$ and $$KS(B^*,A)$$ and used the inner 99% of the resulting D values as a confidence interval for the alternative $$KS(A,B)$$. I.e. $$CI(KS(A,B)) \sim [quantile(KS(A^*,B) \cup KS(B^*,A),.01),quantile(KS(A^*,B) \cup KS(B^*,A),.99)]$$

Finally, if the confidence interval for the alternative lied above the threshold from the null, $$A$$ and $$B$$ were determined to be significantly different (with 99% confidence).

This question came up as the top related result when I was searching for a similar topic, namely, how to compare whether the means of two distributions had the same mean. My question is slightly different because two distributions could be different in ways other than mean (e.g. variance or skew or kurtosis or other "shape measures") but still have nearly equivalent means.

@GregSnow's comment got me on the right track - the term for this is an Equivalence Test.

Another key concept is that you actually cannot compare if two sample distributions are probably the same or not - only if they are close within a bound that you have to pick a priori.

In my case, the Two One-Sided T-Test (TOST) is a recommended procedure and what I used. Once you grok it, it makes sense. It relies on the standard t-test methodology, but uses a trick to make the null hypothesis that the distribution are different (rather than the usual null hypothesis that the distributions are the same).

For my situation, KS was inappropriate because KS DOES test for shape measures, not just means, and I was only interested in means.

Here is the link I used to understand how to implement TOST.

https://www.real-statistics.com/students-t-distribution/equivalence-testing-tost/