1
$\begingroup$

I am comparing few classifiers and I am slightly confused now. I will call the classifiers A, B, C.

If I draw ROC curve, and estimate AUROC, the result points that the classifier A is the best and the classifier C is the worst.

If I draw dependency of the accuracy on the criteria (position of threshold) and integrate the area under the curve, the classifier A is the far best one.

In case of implementation, I can imagine the classifier that is less dependent on criteria (what can be difficult to set in practice) is better than classifier that scores high in AUROC.

My questions:

  1. Does some common metric like "area under accuracy on criteria dependency curve" exists, or it is wrong/useless practice?

  2. Is it possible that this area is not corresponding with AUROC?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This answer assumes a balanced dataset, with as many positive and negative observations. I believe it is enough to show my point.

What does ROC analysis teach us? Let's take a plot of the ROC space: ROC space

ROC analysis tells us that any classifier can be represented by a monotone curve, that goes from the bottom left (when threshold is high and everything is classified as negative) to the top right (when threshold is low and everything is classified as positive).

What about the accuracy? It is maximal at the top left corner (perfect classification) and minimal at the bottom right (100% wrong classification). If your classifier performs no better than random, the ROC curve will follow the diagonal line, and the accuracy will be 50% regardless of the threshold.

Again: all classifiers will need to go from the bottom left to the top right corner, both having 50% accuracy. How can we have a classifier whose accuracy doesn't depend on the threshold?

Easy: follow the diagonal line with a classifier that performs no better than random.

In other words, the accuracy of a useful classifier has to depend on the threshold. So your following statement is incorrect:

In case of implementation, I can imagine the classifier that is less dependent on criteria (what can be difficult to set in practice) is better than classifier that scores high in AUROC.

and therefore an "area under accuracy on criteria dependency curve" would be useless.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot! How do you suggest to measure this dependency? Just by ROC curve? Because ROC produces much different values than "area under accuracy on criteria dependency curve". I would like to use some criteria in my study. And I am afraid the ROC is misleading for practical application - it does not cover the threshold change linearly. $\endgroup$
    – matousc
    Feb 16, 2017 at 8:39
  • $\begingroup$ I suggest this dependency is not useful. Updated my answer to reflect that more precisely. $\endgroup$
    – Calimo
    Feb 16, 2017 at 8:48
  • $\begingroup$ Thanks again, but I still need a bit of clarification. Let us consider threshold in range 0-1. The dependency of accuracy on criteria shows what accuracy I will get exactly with any criteria. Thus the sum of the area under this curve reflect my chances to get best classification (if I pick the threshold "randomly"). I think I cannot say the same think about AUROC - it can describe only part of threshold range (depends on data and ability of classifier). Is it clear? What part of this assumption is not correct? Maybe I miss something basic. $\endgroup$
    – matousc
    Feb 16, 2017 at 9:04
  • $\begingroup$ "Thus the sum of the area under this curve reflect my chances to get best classification" I just showed this is not the case. Also I'm not sure what the "curve" would be here - only that it wouldn't be useful. $\endgroup$
    – Calimo
    Feb 16, 2017 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.