I would like to use two samples Kolmogorov-Smirnov test to check if two given samples are coming from different distributions. For that I use scipy implementation of the KS test.

I have found out that in some cases, when samples are obviously coming from two different distributions the KS does not "see" it (p values is high (around 0.1)). In particular T-test clearly "sees" that means of two distributions are different (p-value around 10^-10). In other words T-tests sees that means of two distributions are different by KS does not sees that the distributions are different.

Bellow is an example reproducing this behaviour. One sample is generated of two Gaussians with means 0 and -10 while another sample is generated with a mixture with means of 0 and 10. The KS test does not "see" the difference between the samples:

import random
from scipy import stats

shift = 10.0
prob  = 0.07

for i in range(20):
    ls1 = [
        random.normalvariate(-shift, 1.0) if random.uniform(0.0, 1.0) < prob
        else random.normalvariate(0.0, 1.0) 
        for i in range(1000)]

    ls2 = [
        random.normalvariate( shift, 1.0) if random.uniform(0.0, 1.0) < prob
        else random.normalvariate(0.0, 1.0) 
        for i in range(1000)]

    ks_2samp = stats.ks_2samp(ls1, ls2).pvalue
    ttest_ind = stats.ttest_ind(ls1, ls2).pvalue

    print ks_2samp, ttest_ind

Here is the output:

0.16580778180902842 9.060856080948288e-12
0.01851544068151054 5.898586305260549e-14
0.00427744491524331 9.7049212240596e-16
0.027130694162290223 1.0689263790159754e-14
0.010125210232304454 3.296184221499672e-14
0.0006280816342195499 1.0177034143495455e-18
0.03264195164443303 1.4076742121406555e-12
0.09090010387130891 1.2443419177675215e-13
0.01851544068151049 2.747214603788716e-15
0.01851544068151049 8.121824308260815e-14
0.00427744491524331 5.6988053773438894e-18
0.006634750962621394 1.3293198764640583e-13
0.05547862104852167 2.8550937455660035e-14
0.2828891626185374 6.495297514372301e-09
0.006634750962621394 3.0442842979678378e-15
0.07742034064789591 3.278591079385276e-12
0.046677646772462215 4.576882344667905e-13
0.00821290761051498 1.0193621244783236e-15
0.02713069416229027 2.6797479930917613e-16
0.012432141728521352 6.9673275270114e-15

I also tried to find two samples (both containing 200 numbers) such that p-value of T-test is as small as possible and p-value of KS test is above 0.06. I found samples where KS test gives p-value = 0.0622 and KS test gives 1.71e-10. This is how the two distributions look like:

enter image description here

And here is the cumulative sum of the ordered elements of two sets:

enter image description here

So, my question is if it is known that KS test fails in some special cases and, if it is the case, what are those cases?

  • $\begingroup$ K-S has not failed it has told you it cannot find evidence that the distributions differ. If you use a technique which has a different purpose and is responsive to different features of the data it may well give different results. That is why we have so many different techniques. $\endgroup$ – mdewey Sep 9 at 15:29
  • $\begingroup$ @mdewey, if K-S test cannot find evidence that that distributions differ in case where they obviously differ, why is it not a failure? If we have a very strong evidence that means of the two distributions differ, why can't we conclude that distributions differ? $\endgroup$ – Roman Sep 9 at 16:02
  • 1
    $\begingroup$ Closely related: stats.stackexchange.com/questions/290767/… $\endgroup$ – Glen_b -Reinstate Monica Sep 10 at 3:12

The Kolmogorov-Smirnov test is a very general omnibus test which compares two distributions against the very general alternative hypothesis that they are different in some respect. The price that is paid for this is that it cannot be very sensitive to a specific hypothesis. By contrast other tests may be very specific against one alternative but not respond to others. Student's t test responds to location shift but is not responsive, for instance, against difference in variance. So the fact that they give different results with a specific data-set is evidence that they are both working exactly as designed and neither is a failure.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.