2
$\begingroup$

How close is my classifier to random guessing?

I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric.

The random guessing line (RGL) from (0,0) to (1,1) has an AUC of 0.5. But so does the blue curve (grey area).

ROC RGL

Wouldn't it be more suitable to use the area between the RGL and the ROC-curve to estimate how "close" a classifier is to actual random guessing?

enter image description here

$\endgroup$
  • $\begingroup$ Have you considered you will be pretty far from the diagonal when you have few data points? $\endgroup$ – Calimo May 7 at 5:31
  • $\begingroup$ *Assuming the data set is big enough, so "no stair steps" and uncertainties from limited sample size. $\endgroup$ – lnathan May 7 at 7:28
  • $\begingroup$ My point being, your measure has a systematic bias on the number of data points. That's not something desirable in general. $\endgroup$ – Calimo May 7 at 9:07
  • 3
    $\begingroup$ The ROC curve should not cross below the diagonal as yours does. This ROC implies there is a problem with your model; it is misspecified in some fundamental way (cf, ROC curve crossing the diagonal). If your curve were always above the diagonal, the area between the ROC & the diagonal would be the same as AUC-.5. Ie, you'd have a different number, but the same information. $\endgroup$ – gung May 7 at 15:25
1
$\begingroup$

First, the 0.5 random guess line is just a visual reference, what we really want to know if how well the classifier performs overall. I'm also not sure how your proposed method would provide a different result, assuming that you subtract first section that is below the 0.5 line.

Second, the AUC also has a nice statistical property where it is equivalent to the Wilcoxon-Mann-Whitney U Test statistic. This is the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

In this case, we don't need to see the AUC value to know this is a poor classifier (and one unlikely to be seen in real life, as it implies that true positive values are consistently rated under a certain probability threshold while false positive values are more evenly distributed).

$\endgroup$
0
$\begingroup$

How close is my classifier to random guessing?

You could test this by using Kolmogorov-Smirnov.

In Python:

from scipy import stats

stats.ks_2samp(actual, preds)

Small K-S statistic and high p-value suggest that your classifier is no better than random.

(Also this might help: How to determine whether a classifier is significantly better than random guessing?)

$\endgroup$
  • $\begingroup$ Does class imbalance affect the k-s statistic? $\endgroup$ – lnathan May 27 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.