# Is Kolmogorov-Smirnov test known to fail in some special cases?

I would like to use two samples Kolmogorov-Smirnov test to check if two given samples are coming from different distributions. For that I use scipy implementation of the KS test.

I have found out that in some cases, when samples are obviously coming from two different distributions the KS does not "see" it (p values is high (around 0.1)). In particular T-test clearly "sees" that means of two distributions are different (p-value around 10^-10). In other words T-tests sees that means of two distributions are different by KS does not sees that the distributions are different.

Bellow is an example reproducing this behaviour. One sample is generated of two Gaussians with means 0 and -10 while another sample is generated with a mixture with means of 0 and 10. The KS test does not "see" the difference between the samples:

import random
from scipy import stats

shift = 10.0
prob  = 0.07

for i in range(20):
ls1 = [
random.normalvariate(-shift, 1.0) if random.uniform(0.0, 1.0) < prob
else random.normalvariate(0.0, 1.0)
for i in range(1000)]

ls2 = [
random.normalvariate( shift, 1.0) if random.uniform(0.0, 1.0) < prob
else random.normalvariate(0.0, 1.0)
for i in range(1000)]

ks_2samp = stats.ks_2samp(ls1, ls2).pvalue
ttest_ind = stats.ttest_ind(ls1, ls2).pvalue

print ks_2samp, ttest_ind


Here is the output:

0.16580778180902842 9.060856080948288e-12
0.01851544068151054 5.898586305260549e-14
0.00427744491524331 9.7049212240596e-16
0.027130694162290223 1.0689263790159754e-14
0.010125210232304454 3.296184221499672e-14
0.0006280816342195499 1.0177034143495455e-18
0.03264195164443303 1.4076742121406555e-12
0.09090010387130891 1.2443419177675215e-13
0.01851544068151049 2.747214603788716e-15
0.01851544068151049 8.121824308260815e-14
0.00427744491524331 5.6988053773438894e-18
0.006634750962621394 1.3293198764640583e-13
0.05547862104852167 2.8550937455660035e-14
0.2828891626185374 6.495297514372301e-09
0.006634750962621394 3.0442842979678378e-15
0.07742034064789591 3.278591079385276e-12
0.046677646772462215 4.576882344667905e-13
0.00821290761051498 1.0193621244783236e-15
0.02713069416229027 2.6797479930917613e-16
0.012432141728521352 6.9673275270114e-15


I also tried to find two samples (both containing 200 numbers) such that p-value of T-test is as small as possible and p-value of KS test is above 0.06. I found samples where KS test gives p-value = 0.0622 and KS test gives 1.71e-10. This is how the two distributions look like:

And here is the cumulative sum of the ordered elements of two sets:

So, my question is if it is known that KS test fails in some special cases and, if it is the case, what are those cases?

• K-S has not failed it has told you it cannot find evidence that the distributions differ. If you use a technique which has a different purpose and is responsive to different features of the data it may well give different results. That is why we have so many different techniques. – mdewey Sep 9 at 15:29
• @mdewey, if K-S test cannot find evidence that that distributions differ in case where they obviously differ, why is it not a failure? If we have a very strong evidence that means of the two distributions differ, why can't we conclude that distributions differ? – Roman Sep 9 at 16:02
• Closely related: stats.stackexchange.com/questions/290767/… – Glen_b -Reinstate Monica Sep 10 at 3:12