0
$\begingroup$

I have 2 binary classifiers and a test set.

For the first of the classifiers I can compute any metrics for any value of a threshold, e.g. I can plot ROC curve and calculate precision, recall, F1 etc at any point of the curve.

For the second classifier the value of the threshold is fixed, e.g. I can calculate all the same metrics but only for one point of ROC curve.

Is there a way to say something about which classifier is better under these conditions? For example, would the fact, that the single dot computed for the second classifier is above ROC curve of the first classifier, mean that the second classifier is better overall?

$\endgroup$
  • $\begingroup$ How did you build that second classifier? $\endgroup$ – David Mar 18 '19 at 8:53
  • $\begingroup$ I haven't, it's available as a black box $\endgroup$ – Mendor Mar 18 '19 at 19:53
  • $\begingroup$ Why on Earth would you use a method that you have no control over? $\endgroup$ – David Mar 20 '19 at 7:07
  • $\begingroup$ Thanks for your feedback :) $\endgroup$ – Mendor Mar 21 '19 at 15:09
1
$\begingroup$

For the first classifier, you have the option to tune. There may be a point on ROC curve, in which all your metrics are superior than the first one. Or, conversely, the second classifier might overperform the first one in any configuration. Having all metrics superior for one of the classifiers leaves you with a simple choice. Otherwise, you need to decide on which metric you want to care for much, and choose the classifier that is superior. Eventually, it boils down to model selection. And, I’m assuming you calculate these on some sort of validation set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.