Comparing two binary classifiers

Question

I have 2 binary classifiers and a test set.

For the first of the classifiers I can compute any metrics for any value of a threshold, e.g. I can plot ROC curve and calculate precision, recall, F1 etc at any point of the curve.

For the second classifier the value of the threshold is fixed, e.g. I can calculate all the same metrics but only for one point of ROC curve.

Is there a way to say something about which classifier is better under these conditions? For example, would the fact, that the single dot computed for the second classifier is above ROC curve of the first classifier, mean that the second classifier is better overall?

Answer 1

For the first classifier, you have the option to tune. There may be a point on ROC curve, in which all your metrics are superior than the first one. Or, conversely, the second classifier might overperform the first one in any configuration. Having all metrics superior for one of the classifiers leaves you with a simple choice. Otherwise, you need to decide on which metric you want to care for much, and choose the classifier that is superior. Eventually, it boils down to model selection. And, I’m assuming you calculate these on some sort of validation set.