6
$\begingroup$

I have preformed two-sided Kolmogorov-Smirnov test pair-wise across 6 datasets. It performs as expected (p-value <<< 0.05 in most cases, p-value=1 for identity), except for one pair where it assigns the p-value == 1 for two very distinct distributions as shown below:

Comparison of the two distributions

The results of conducting the test are

KS test less: statistic=0.0, pvalue=1.0
KS test greater: statistic=0.7213042381292454, pvalue=0.0
KS test two-sided: statistic=0.7213042381292454, pvalue=1.0

The results for less and greater make sense to me, but I can't figure out why would the two-sided return p-value = 1, since it should be possible to reject that sample1 and sample2 are drawn from the same distribution?

Edit: The code calculating the metric is trivial:

from scipy import stats
for alternative in ['less', 'greater', 'two-sided']:
    ks_stat, p_value = stats.ks_2samp(sample1, sample2, alternative=alternative)
    print(f"KS test {alternative}: statistic={ks_stat}, pvalue={p_value}")

Edit 2 Subsampling fixes the issue, so there's a problem with the sample set itself it seems:

from scipy import stats for alternative in ['less', 'greater', 'two-sided']:
    ks_stat, p_value = stats.ks_2samp(sample_a.sample(100), sample_b.sample(100), alternative=alternative)
    print(f"KS test {alternative}: statistic={ks_stat}, pvalue={p_value}")

Yields:

KS test less: statistic=0.0, pvalue=1.0
KS test greater: statistic=0.88, pvalue=6.745216854681774e-41
KS test two-sided: statistic=0.88, pvalue=1.349043370936356e-40

Edit 3: Switching for asymptotic method also seems to fix the issue:

KS test exact: statistic=0.7213042381292454, pvalue=1.0
KS test asymp: statistic=0.7213042381292454, pvalue=0.0
KS test auto: statistic=0.7213042381292454, pvalue=1.0
$\endgroup$
6
  • 2
    $\begingroup$ Perhaps it may be helpful if you could provide code and data. $\endgroup$
    – laurab
    Jun 5, 2023 at 12:28
  • $\begingroup$ I've added the code, but it's trivial call of the function, hence I originally omitted it. The statistical summary of the data is in the plot. Is there any feature of the data you think I should add? The dataset is quite big, I'd have to put it on external hosting to share it in full. $\endgroup$ Jun 5, 2023 at 12:48
  • $\begingroup$ If the problem is a bug in the software or a misinterpretation of what it does, you should be able to reproduce it with a tiny dataset. Three values in each group ought to do it. $\endgroup$
    – whuber
    Jun 5, 2023 at 13:04
  • 1
    $\begingroup$ Fair point - I've subsampled 100 samples (see edit 2) and that fixes the issue, so it seems that it's not a problem with my interpretation of the function. $\endgroup$ Jun 5, 2023 at 13:37
  • 1
    $\begingroup$ Actually it seems that the issue is resolved if I switch on the asymptotic mode, see the edit 3. That's a surprise, I expected the exact mode to be the more precise one. $\endgroup$ Jun 5, 2023 at 17:35

1 Answer 1

2
$\begingroup$

You are correct that the p-value for the two-sided test should be tiny in this case, so this looks like a software bug to me. The fact that the problematic case occurs with a calculated p-value of p-value = 1 also suggests that it is a bug at a border case (which happens in programming sometimes). It is good that switching to the asymptotic method fixes the problem, but it would still be best if the main method also did the calculation properly. I recommend submitting it as a bug to the package maintainer and letting them sort out the problem in the underlying code.

$\endgroup$
1
  • $\begingroup$ Ok, thank you for the summary, I'll try to submit a report. $\endgroup$ Jun 7, 2023 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.