I am trying to interpret the Kolmogorov–Smirnov test in the case of predictive analytics to compare three models: a neural network, a decision tree and a logistic regression. The target variable is binary: 0=bads, 1=goods. According to the KS theοry, I compare the cumulative distribution of goods with the cumulative distribution of bads to check whether they come from the same distribution, so I have a null and an alternative hypothesis.

In books about predictive analytics, the comparison among the three predictive models is done according to how large the KS statistic is. So the model with the largest KS statistic is chosen. Though software packages related to predictive analytics and data mining in general do not report the p-value of the KS statistics but only its size. But since the KS is a hypothesis test, the biggest KS statistic might not be significant, although it might be the largest related to the three predictive models.

So the question is: what is the point in checking only the size of the KS statistic, as the predictive analytics books and software do, without checking the p-value.

  • 1
    $\begingroup$ Could you clarify how exactly the KS statistic is used? It's not clear in your question. $\endgroup$
    – Glen_b
    Feb 2 '16 at 8:00
  • $\begingroup$ Doesn't the two-sample KS require a continuous distributions? Or are you comparing predictions? $\endgroup$
    – dimitriy
    Jun 19 '20 at 6:19
  • 1
    $\begingroup$ It's my understand that the names Kolmogorov and Smirnov end with the same letter in Russian, so you can transliterate both ending -ov (which seems now universal in English-language literature I see) or you can transliterate both ending -off, which is very old-fashioned, but you should transliterate the same way. $\endgroup$
    – Nick Cox
    Jun 22 '21 at 16:05

There are some serious known issues with the KS test's power. Essentially, it can reject the null hypothesis many times when it shouldn't. But the test statistic itself still gives a good measurement for distances between distributions. So even if the p-value ends up being not useful, the test statistic is still a good distance measurement (also, p-values themselves don't really mean anything useful, but that's a different story...).

  • $\begingroup$ This is a feature, not a bug, of hypothesis testing. If the distributions are different, the KS test absolutely should catch that the distributions are different. If that difference is minuscule and not of practical importance, the test still should catch that difference, and missing that difference is a type II error. $\endgroup$
    – Dave
    Jun 22 '21 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.