Kolmogorov Smirnoff and Predictive Analytics

Question

I am trying to interpret the Kolmogorof Smirnoff test in the case of predicitve analytics to compare three models: a neural network, a decision tree and a logistic regression. The target variable is binary: 0=bads, 1=goods. According to the KS theοry i compare the cumulative ditribution of goods with the cumulative distribution of bads to check whether they come from the same distribution, so i have a null and an alternative hypothesis.

In books about predicitve analytics the comparison among the three predicitve models is done according to how large the KS statistic is. So the model with the largest KS statistic is chosen. Though software packages related to predicitve analytics and data mining in general do not report the p-value of the KS statistics but only its size. But since the KS is a hypothesis test, the biggest KS statistic might not be significant, although it might be the largest related to the three predictive models.

So the question is what is the point in checking only the size of the KS statistic, as the predicitve analytics books and software do, without checking the p-value.

Thanks in advance,

Andreas

Answer 1

There are some serious known issues with the KS test's power. Essentially, it can reject the null hypothesis many times when it shouldn't. But the test statistic itself still gives a good measurement for distances between distributions. So even if the p-value ends up being not useful, the test statistic is still a good distance measurement (also, p-values themselves don't really mean anything useful, but that's a different story...).