2
$\begingroup$

I have a binary and balanced dataset. Do I have to see the AUROC as the different trade-offs between the TPR and the FPR and the accuracy as a result with a threshold of 0.5? When is accuracy a better metric than AUROC?

$\endgroup$
  • 1
    $\begingroup$ Accuracy, TPR, FPR, and AUROC measure different things. Apparently (I haven't been able to trace the source and see the proof), AUROC estimates the probability that the classifier will give an observation in class 1 (or positive or however it's coded) a higher score than an observation in class 0 (or negative or whatever). Depending on your use case, that might be more or less useful than accuracy, TPR, FPR, or other such statistics. $\endgroup$ – Dan Hicks Aug 12 '17 at 22:08
  • 1
    $\begingroup$ Given that your model will be most likely far from perfect, what should the model be able to do best when you evaluate it? The "best" metric will be able to address exactly this question. $\endgroup$ – g3o2 Aug 12 '17 at 22:47
3
$\begingroup$

AUROC helps you to determine what threshold you must choose for your model. That will depend on the impact of errors, both FP and FN, in your problem.

You need to understand how your model is going to be used and the implications of the deployment strategy you choose.

$\endgroup$
  • 1
    $\begingroup$ This means, for example, that if I'm intersted in minimizing the cost of decisions based on the classifier, I should go for higher AUROC and select the most appropriate threshold, right? $\endgroup$ – Franco Jun 6 '18 at 19:28
  • 1
    $\begingroup$ @Franco, yes, that's the implication. $\endgroup$ – gung - Reinstate Monica Jun 7 '18 at 1:13
  • $\begingroup$ A higher AUC is not always what you want, in some problem you may only care about the behaviour at the low end or high end, that is the top-right corner or the bottom-left corner. $\endgroup$ – Picarus Jun 14 '18 at 3:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.