2
$\begingroup$

I am trying to interpret the Kolmogorof Smirnoff test in the case of predicitve analytics to compare three models: a neural network, a decision tree and a logistic regression. The target variable is binary: 0=bads, 1=goods. According to the KS theοry i compare the cumulative ditribution of goods with the cumulative distribution of bads to check whether they come from the same distribution, so i have a null and an alternative hypothesis.

In books about predicitve analytics the comparison among the three predicitve models is done according to how large the KS statistic is. So the model with the largest KS statistic is chosen. Though software packages related to predicitve analytics and data mining in general do not report the p-value of the KS statistics but only its size. But since the KS is a hypothesis test, the biggest KS statistic might not be significant, although it might be the largest related to the three predictive models.

So the question is what is the point in checking only the size of the KS statistic, as the predicitve analytics books and software do, without checking the p-value.

Thanks in advance,

Andreas

$\endgroup$
  • $\begingroup$ Could you clarify how exactly the KS statistic is used? It's not clear in your question. $\endgroup$ – Glen_b -Reinstate Monica Feb 2 '16 at 8:00
0
$\begingroup$

There are some serious known issues with the KS test's power. Essentially, it can reject the null hypothesis many times when it shouldn't. But the test statistic itself still gives a good measurement for distances between distributions. So even if the p-value ends up being not useful, the test statistic is still a good distance measurement (also, p-values themselves don't really mean anything useful, but that's a different story...).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.