30
$\begingroup$

Let's say I have two samples. If I want to tell whether they are pulled from different populations, I can run a t-test. But let's say I want to test whether the samples are from the same population. How does one do this? That is, how do I calculate the statistical probability that these two samples were pulled from the same population?

$\endgroup$
  • 1
    $\begingroup$ Please explain--as quantitatively as possible--what you mean by "same." It would help to clarify what you mean by "condition," too. $\endgroup$ – whuber Jan 23 '14 at 20:52
  • $\begingroup$ A test like a two-sample Kolmogorov-Smirnov (not the only possibility; with the usual assumptions, the t-test is testing the same thing, as you note) can test whether the population distributions are different (but failure to reject doesn't mean they're actually the same). However, no test can tell you whether two distributions that are not too dissimilar are actually from the same population, rather than two different populations with similar distributions. That would have to come either from assumptions or other investigation. ...ctd $\endgroup$ – Glen_b Jan 23 '14 at 22:56
  • 3
    $\begingroup$ ctd ... Similarly, tests can't even tell you that the distributions are identical, since they can differ in trivial ways. You may want to search on 'equivalence test' or 'equivalence testing', on which you should get quite a few hits here, or on google. $\endgroup$ – Glen_b Jan 23 '14 at 23:00
20
$\begingroup$

The tests that compare distributions are rule-out tests. They start with the null hypothesis that the 2 populations are identical, then try to reject that hypothesis. We can never prove the null to be true, just reject it, so these tests cannot really be used to show that 2 samples come from the same population (or identical populations).

This is because there could be minor differences in the distributions (meaning they are not identical), but so small that tests cannot really find the difference.

Consider 2 distributions, the first is uniform from 0 to 1, the second is a mixture of 2 uniforms, so it is 1 between 0 and 0.999, and also 1 between 9.999 and 10 (0 elsewhere). So clearly these distributions are different (whether the difference is meaningful is another question), but if you take a sample size of 50 from each (total 100) there is over a 90% chance that you will only see values between 0 and 0.999 and be unable to see any real difference.

There are ways to do what is called equivalence testing where you ask if the 2 distributions/populations are equivalent, but you need to define what you consider to be equivalent. It is usually that some measure of difference is within a given range, i.e. the difference in the 2 means is less than 5% of the average of the 2 means, or the KS statistic is below a given cut-off, etc. If you can then calculate a confidence interval for the difference statistic (difference of means could just be the t confidence interval, bootstrapping, simulation, or other methods may be needed for other statistics). If the entire confidence interval falls in the "equivalence region" then we consider the 2 populations/distributions to be "equivalent".

The hard part is figuring out what the equivalence region should be.

$\endgroup$
  • 2
    $\begingroup$ A null hypothesis test can never provide evidence for the null hypothesis, true. Model selection, either Bayesian or based on some "criterion" (AIC, BIC) could however indicate that a null model (identical distributions) is a better description of the data than an alternative model (different distribution). All that under a bunch of assumptions of course. $\endgroup$ – A. Donda Feb 4 '15 at 2:23
6
$\begingroup$

http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

Assuming your sample values come from continuous distributions, I would suggest the Kolmogorov-Smirnov test. It can be used to test whether two samples come from different distributions (this is how I am interpreting your usage of population) based on their associated empirical distributions.

Directly from Wikipedia:

The null distribution of this statistic is calculated under the null hypothesis that the samples are drawn from the same distribution (in the two-sample case)

The ks.test function in R can be used for this test.

While it is true the kstest does not test for homogeneity, I would argue that if you fail to reject with a large enough sample size (a high powered test), you can claim the differences are not practically significant. You could infer that if differences do exist, they are likely not meaningful (again, assuming large sample size). You cannot conclude they are from the same population as others have correctly stated. All this being said, typically I would just graphically examine the two samples for similarity.

$\endgroup$
  • 6
    $\begingroup$ I doubt that KS test can be used to show distributional equivalence. $\endgroup$ – Michael M Jan 23 '14 at 22:35
  • $\begingroup$ @MichaelMayer that's exactly right. OP is interested in a test of homogeneity... which has many underlying methodological issues. KS for heterogeneity also has its issues: practically, it will reject in large samples regardless of whether populations are practically identical in all aspects. It just goes to show that testing, and consequently p-values, are better conceived of as measures of sample size than statistical significance. $\endgroup$ – AdamO Jan 24 '14 at 3:33
  • $\begingroup$ @AdamO Yes, but if you have large samples and you fail to reject, I would be confident that the populations are practically identical. As far as I know, there is no theory to back this up, but from experience, knowing that the KS for heterogeneity can detect minute differences with large sample size can allow you to use a failed large sample test as a de facto statement of practically identical populations. Does my response answer the question "calculate the statistical probability that these two samples were pulled from the same population"? Certainly not. $\endgroup$ – Underminer Apr 29 '14 at 20:52
  • $\begingroup$ What can I do if my points are two-dimensional? That is, I have two samples of two-dimensional points, and I want to know if they come from distinct distributions. $\endgroup$ – becko Oct 28 '17 at 19:12
  • $\begingroup$ The KS test only works against a pre-defined distribution, not from a distribution with parameters estimated from data. $\endgroup$ – qwr Apr 5 at 14:55
1
$\begingroup$

You can use a 'shift function' which checks whether the 2 distributions differ at at each decile. While its technically a test of whether they are from different populations rather than the same, if the distributions don't differ on any of the deciles then you can be reasonably sure they are from the same population, especially if the group sizes are large.

I would also visualize the 2 groups: overlay their distributions and see if they resemble each other, or better yet draw a couple of thousand bootstrap samples from each group and plot those, as this would give you an idea of whether they come from the same population particularly if the population in question isnt normally distributed for you given variable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.