I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve.
But, after this is computed, only one threshold is actually chosen for classifying negative and positive examples.
So, I wonder why I've often seen AUC's compared, when comparing two ML algorithms?
Wouldn't it be better to compare, for example, the
F-scores of the BEST threshold from each ROC curve from the two curves?
And, as a follow up, does the classifier associated with the better threshold always mean that it has a higher AUC score? I think I know of some counter examples....but wondering if someone can shed light on this.