4
$\begingroup$

How close is my classifier to random guessing?

I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric.

The random guessing line (RGL) from (0,0) to (1,1) has an AUC of 0.5. But so does the blue curve (grey area).

ROC RGL

Wouldn't it be more suitable to use the area between the RGL and the ROC-curve to estimate how "close" a classifier is to actual random guessing?

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ Have you considered you will be pretty far from the diagonal when you have few data points? $\endgroup$
    – Calimo
    May 7, 2019 at 5:31
  • $\begingroup$ *Assuming the data set is big enough, so "no stair steps" and uncertainties from limited sample size. $\endgroup$
    – Laksan Nathan
    May 7, 2019 at 7:28
  • $\begingroup$ My point being, your measure has a systematic bias on the number of data points. That's not something desirable in general. $\endgroup$
    – Calimo
    May 7, 2019 at 9:07
  • 3
    $\begingroup$ The ROC curve should not cross below the diagonal as yours does. This ROC implies there is a problem with your model; it is misspecified in some fundamental way (cf, ROC curve crossing the diagonal). If your curve were always above the diagonal, the area between the ROC & the diagonal would be the same as AUC-.5. Ie, you'd have a different number, but the same information. $\endgroup$
    – gung - Reinstate Monica
    May 7, 2019 at 15:25

2 Answers 2

Reset to default
2
$\begingroup$

First, the 0.5 random guess line is just a visual reference, what we really want to know if how well the classifier performs overall. I'm also not sure how your proposed method would provide a different result, assuming that you subtract first section that is below the 0.5 line.

Second, the AUC also has a nice statistical property where it is equivalent to the Wilcoxon-Mann-Whitney U Test statistic. This is the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

In this case, we don't need to see the AUC value to know this is a poor classifier (and one unlikely to be seen in real life, as it implies that true positive values are consistently rated under a certain probability threshold while false positive values are more evenly distributed).

Improve this answer
$\endgroup$
0
$\begingroup$

How close is my classifier to random guessing?

You could test this by using Kolmogorov-Smirnov.

In Python:

from scipy import stats

stats.ks_2samp(actual, preds)

Small K-S statistic and high p-value suggest that your classifier is no better than random.

(Also this might help: How to determine whether a classifier is significantly better than random guessing?)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Does class imbalance affect the k-s statistic? $\endgroup$
    – Laksan Nathan
    May 27, 2019 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.