2
$\begingroup$
from scipy import stats

stats.ks_2samp([ 0.09434497,0.07581871,0.03280419,0.07415525,0.07232668,0.10093938,
  0.10348316,0.06529852,0.10967376,0.07681485,0.00863077,0.10323866,
  0.08247111],[ 0.05823918,0.09467191,0.04121951,0.13934854,0.07375773,0.06355782,
  0.13376396,0.08190715,0.05770731,0.07871597,0.03935798,0.04946341,
  0.08828953])

stats.ks_2samp([ 0.05422732,0.08636203,0.04473298,0.14424102,0.08507371,0.05459249,
  0.13365118,0.08124969,0.05057565,0.08800528,0.0478369,0.04308972,
  0.08636203],
[ 0.06545017,0.0973833,0.03399041,0.05891224,0.07687014,0.08660129,
  0.06096384,0.09967657,0.05154278,0.06509205,0.05573505,0.10797274,
  0.13980942])

output

Ks_2sampResult(statistic=0.3076923076923077, pvalue=0.48890821513441646) Ks_2sampResult(statistic=0.3076923076923077, pvalue=0.48890821513441646)

Doesn't make sense... anyone understand why?

$\endgroup$
  • $\begingroup$ You perhaps need to share the reason why you're expecting different KS statistics in the two cases. If it's just that the exact coincidence seems spooky, consider how many distinct values are possible when comparing two samples of 13 observations. $\endgroup$ – Scortchi - Reinstate Monica Nov 12 '18 at 1:08
  • $\begingroup$ It happens with multiple samples, even randomly generated. I am not sure if the implementation is wrong. But the statistics should be different for 4 completely different arrays right? $\endgroup$ – user99355 Nov 12 '18 at 1:51
2
$\begingroup$

I thought I'd answered a question like this before but I can't seem to locate it.

All your samples have n=13.

Hence any difference in ecdf will be an integer multiple of 1/13: 0/13, 1/13, 2/13 ... (the numerator will simply be a largest difference in counts derived from a function of the ranks of the combined samples)

Consequently, that two pairs of observations will have a largest difference that's got the same small-integer numerator is not remotely surprising -- and they both do, it's 4/13.

[However, if your python library is assigning the same p-value to both instances without warning you, that's a concern, since you have tied ranks in one of the samples, which should impact the null distribution.]

$\endgroup$
  • $\begingroup$ Thanks so much. Makes sense. What is the limit after which this doesn't happen? $\endgroup$ – user99355 Nov 12 '18 at 7:04
  • $\begingroup$ There isn't one. The two sample test is always discrete. Any time you have the same pairs of sample sizes in both sets of samples, you could get the same fraction for the difference. It can even happen with different $n_1,n_2$ pairs with suitable common factors. Of course as $n_1,n_2$ both become large, the chance of getting the same fraction will diminish, but will remain non-zero $\endgroup$ – Glen_b -Reinstate Monica Nov 12 '18 at 7:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.