1
$\begingroup$

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve.

But, after this is computed, only one threshold is actually chosen for classifying negative and positive examples.

So, I wonder why I've often seen AUC's compared, when comparing two ML algorithms?

Wouldn't it be better to compare, for example, the F-scores of the BEST threshold from each ROC curve from the two curves?

And, as a follow up, does the classifier associated with the better threshold always mean that it has a higher AUC score? I think I know of some counter examples....but wondering if someone can shed light on this.

$\endgroup$
4
$\begingroup$

There is no given best threshold. It depends on your cost of false-negative and false-positive and your plan of operation.

Example: Some model classifying either GOOD or BAD investments could be applied with different thresholds when used by a banker or venture capitalists. The careful banker would prioritize to have a very low FALSE-GOOD to never loose money. The daring venture capitalist would accept a higher FALSE-GOOD to avoid ever missing out on big investment opportunities(FALSE-BAD).

ROC-plot shows the "sensitivity" vs "1-specificity" for any possible threshold. The AUC could be understood as summarizing the 'average' classification performance over any threshold. AUC is of course not an optimal metric if you already have chosen one specific threshold.

In practice you would probably:

  • build a model first
  • investigate your cross validated ROC-plot
  • wonder how to utilize such predictions
  • decide for a hard threshold (or stay in probability land)
$\endgroup$
  • $\begingroup$ Right, so at some point you have to decide on a "hard threshold" , as you said. So, if I'm comparing two models, wouldn't it be best to: 1) determine a measurement (say, F measure, with beta = 2). 2) Find a threshold for each model that maximizes that measurement. 3) Finally, pick the model that has a better F measure. I don't understand the need for an AUC to compare these unless I'm missing something....? $\endgroup$ – Candic3 Feb 20 '16 at 19:08
  • 1
    $\begingroup$ well yes... AUC is most useful for comparison when no obvious single threshold is given.. Instead of one metric such as F measure, you could try to estimate your practical costs and gains of acting on predictions by a certain threshold by a certain model. That could possibly answer which model is best for your purpose. You could also proceed by combining both models into an ensemble $\endgroup$ – Soren Havelund Welling Feb 21 '16 at 12:36
  • 1
    $\begingroup$ I would also add that thresholds are not always chosen. Sometimes what matters is the scores ability to rank the target variable. In such a case, AUC is very informative as the whole distribution can be observed in one plot. $\endgroup$ – Zelazny7 Feb 22 '16 at 20:12
  • $\begingroup$ Hm. I'm not sure I follow that @Zelazny7, do you mind elaborating? Perhaps you're suggesting something like computing risk using a binary classifier? And, one could use AUC for determining how well we rank the risk of the examples in our classifier. Is that true? $\endgroup$ – Candic3 Feb 24 '16 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.