3
$\begingroup$

In practice, there can be a classifier that gives far better performance at a specific acceptable threshold than an "optimal" classifier with better average performance across range of thresholds (higher AUC) but not so much at that threshold.

For example:

  • Classifier 1: 80% TPR with 5% FPR, 95% TPR with 40% FPR, AUC = 0.6
  • Classifier 2: 40% TPR with 5% FPR, 95% TPR with 20% FPR, AUC = 0.9

Shouldn't I use classifier 1 instead of classifier 2 if I am operating around 5% FPR acceptable threshold?

Also, what if I am allowed to run near 10% FPR?

  • Should I just check TPR from both classifiers corresponding to 10% FPR and pick the classifier that has higher TPR?
  • Or compute a partial area under curve until 10% FPR and pick the one with the highest?
$\endgroup$
1
$\begingroup$

This is actually exactly the problem that ROC curves were originally constructed to solve! The basic idea of a classifier that returns some "confidence" score (like a posterior predictive probability) is that by adjusting the threshold that you use to take an action, you directly influence the TPR and the FPR for your procedure.

A ROC curve is a diagram that explicitly shows the tradeoff: if you accept a higher FPR, you'll get a higher TPR. If you're constrained to only accept at most a specific FPR, then the only relevant statistic is the TPR at that FPR. So if your particular application can only accept at most some FPR, then you should choose a threshold that corresponds to that FPR. Full stop.

This arises in all sorts of real-world situations. If the cost of a false positive is high, you'll want to have a lower FPR.

So if you're constrained to only accept a specific FPR, your choices are obvious: either you pick the model with the highest TPR at that FPR, or you choose a model with a lower TPR. Clearly, any lower TPR is (probably*) worse, because you'll achieve fewer true positives at that FPR.

* The one caveat here is that FPR and TPR are statistics, and like every other statistic, they are subject to random variation. The only factor that you can control is the choice of threshold. The FPR and TPR at that threshold are naturally estimated with error, so the best practice would be to estimate the error bounds on TPR and FPR at your choice of threshold and then make the comparison. This is easy enough to do, since all of these statistics are binomial proportions, and tests of binomial proportions are well-studied.

$\endgroup$
  • $\begingroup$ Thanks. I am following up on an old answer from you: So if I know that FPR is 10% with a margin of error +- 2%, then should I look at the partial area under curve between 8% and 12% and pick the classifier that has most AUC under that limited range of FPR? $\endgroup$ – toing Aug 30 '18 at 15:24
  • $\begingroup$ No. Suppose your largest acceptable FPR is $10%$, but the MOE at $10\%$ is $\pm2%$. You pick the threshold that corresponds to an upper bound of 10%, which works out to 8%. You do this because otherwise there's a rather high probability that the FPR you will see in reality will exceed your maximum acceptable FPR. $\endgroup$ – Sycorax Aug 30 '18 at 15:27
  • $\begingroup$ Got it - and then should I look up partial AUC between 0% and 8% FPR of various classifiers and pick the one with the largest area in that range in this specific example? $\endgroup$ – toing Aug 31 '18 at 0:45
  • $\begingroup$ I'm not sure that area is really what you want. What you want is the highest TPR; so just look at the TPR at 8% FPR. $\endgroup$ – Sycorax Aug 31 '18 at 1:00
  • $\begingroup$ Understood. If I were to use that partial area, will it give me a measure of average performance if the classifier was to operate with FPR in the range of 0% and 8% instead of at 8%? $\endgroup$ – toing Aug 31 '18 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.