I have found out that in some cases, when samples are obviously coming from two different distributions the KS does not "see" it (p values is high (around 0.1)). In particular T-test clearly "sees" that means of two distributions are different (p-value around
10^-10). In other words T-tests sees that means of two distributions are different by KS does not sees that the distributions are different.
Bellow is an example reproducing this behaviour. One sample is generated of two Gaussians with means 0 and -10 while another sample is generated with a mixture with means of 0 and 10. The KS test does not "see" the difference between the samples:
import random from scipy import stats shift = 10.0 prob = 0.07 for i in range(20): ls1 = [ random.normalvariate(-shift, 1.0) if random.uniform(0.0, 1.0) < prob else random.normalvariate(0.0, 1.0) for i in range(1000)] ls2 = [ random.normalvariate( shift, 1.0) if random.uniform(0.0, 1.0) < prob else random.normalvariate(0.0, 1.0) for i in range(1000)] ks_2samp = stats.ks_2samp(ls1, ls2).pvalue ttest_ind = stats.ttest_ind(ls1, ls2).pvalue print ks_2samp, ttest_ind
Here is the output:
0.16580778180902842 9.060856080948288e-12 0.01851544068151054 5.898586305260549e-14 0.00427744491524331 9.7049212240596e-16 0.027130694162290223 1.0689263790159754e-14 0.010125210232304454 3.296184221499672e-14 0.0006280816342195499 1.0177034143495455e-18 0.03264195164443303 1.4076742121406555e-12 0.09090010387130891 1.2443419177675215e-13 0.01851544068151049 2.747214603788716e-15 0.01851544068151049 8.121824308260815e-14 0.00427744491524331 5.6988053773438894e-18 0.006634750962621394 1.3293198764640583e-13 0.05547862104852167 2.8550937455660035e-14 0.2828891626185374 6.495297514372301e-09 0.006634750962621394 3.0442842979678378e-15 0.07742034064789591 3.278591079385276e-12 0.046677646772462215 4.576882344667905e-13 0.00821290761051498 1.0193621244783236e-15 0.02713069416229027 2.6797479930917613e-16 0.012432141728521352 6.9673275270114e-15
I also tried to find two samples (both containing 200 numbers) such that p-value of T-test is as small as possible and p-value of KS test is above
0.06. I found samples where KS test gives p-value = 0.0622 and KS test gives
1.71e-10. This is how the two distributions look like:
And here is the cumulative sum of the ordered elements of two sets:
So, my question is if it is known that KS test fails in some special cases and, if it is the case, what are those cases?