How is Kolmogorov Smirnov a good measure of classifier performance?

Question

KS tells how different two distributions are from each other. Assume I have a classifier model that has completely misunderstood the data and predicts a 0 for every 1 and a 1 for every 0. Assume the probability distributions are really far apart.

Wouldn't this model have a horrible accuracy and a really good KS, since KS doesn't measure accuracy but only distribution difference? If I measure my model per KS it says I will have an excellent model when it is garbage at correctly predicting the output.

What am I not understanding in regards to using KS to evaluate classifier performance?

EDIT: I am asking because I've encountered KS as measure of model performance in online articles and professional settings

Answer 1

The Kolmogorov-Smirnoff statistic has some problems as a machine learning performance metric that are worth knowing.

First, KS only assesses the maximum vertical distance between CDFs. Consequently, it will not be sensitive to differences elsewhere that a metric like Brier score or log loss will detect. Whenever either of those metrics encounter a prediction that differs from the observed value, there is a penalty.

Second, as you have noted, the KS test misses reverse calibration. If my predictions are consistently the opposite of what they should be, there are ways to handle that, but I need to know, not be deceived into thinking my predictions are good.

A combination of a poor score on a metric that catches reverse calibration and a high score on KS (so a low p-value) could be a signal that there is reverse calibration. I suspect, however, there to be considerably better alternatives.