I've seen that people are sometimes using Kolmogorov-Smirnov statistic to determine a cutoff in a binary classification models, such as logistic regression. However, i do not fully understand why and what are the implications of such a choice.

So far i've seen two different ways of using KS to choose a cutoff for binary classifier. One is to compare the empirical distribution function of predictions for class 1 and class 0 and place a cutoff where the difference between the EDFs' values is the biggest (as measured by the KS statistic)

This is show in Figure 3 of this paper by Van Gool et al., 2012 or it is mentioned in this paper on measuring the quality of credit scoring models.

Another approach i've seen is to compare the TPR and FPR using KS statistic as done in this document.

I haven't seen a thorough explanation why exactly would one want to choose the cutoff this way? How does this compare to other ways of choosing a cutoff (such as based on the share of class 1 and 0 on all observations? - i.e. if 30% of past observations were class 1, then i would choose to predict class one for the highest 30% of my predictions).

My intuitive understanding is that if i choose a cutoff where the difference between the two EDFs is biggest, then i somehow choose a point such that to the left of it i am adding to group 0 True negatives faster than False negatives (the gap between the two EDFs is opening up),whereas to the right of the chosen point i am adding True positives to group 1 faster than i am adding False positives to it (the gap between the curves is closing down).

Is my intuition correct? Is there something else i can say about the cutoff? How about a case when the gap between the two curves would first open up, then close, then open up and close again? Would still a cutoff based on KS statistic make sense? (Yes, i know that "whether the choice of cutoff makes sense depends on the costs coming from bad classification").



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.