1
$\begingroup$

Clearly, Brier Score and AUROC are better performance measures to compare classifiers. However, besides that, I am interested in a let's call it more economic view. I could imagine a classifier being much more accurate than the other. Nevertheless, the other classifier dominates in the ROC space for some sensitivity and specificity values. In introducing costs for misclassification, I might prefer exactly these values since the overall costs will be lower. I would like to point that out in the master thesis I'm writing right now. Is this correct and a legitimate point to make?

$\endgroup$
2
  • $\begingroup$ It is not clear to me that Brier score and AUROC are "better". I think it depends on what you are trying to do with the classifier. $\endgroup$
    – Peter Flom
    Commented Jan 2, 2016 at 12:13
  • $\begingroup$ @ Peter Flom. I guess, I wasn't so clear about that. But when I'm comparing two classifiers in general (threshold-independent), sensitivity and specificity are not well-suited since maybe for one classifier I get higher sensitivity/specificity for a threshold of 0.5 and for another one I get better values for 0.4. This is why I used AUROC and Brier Score. $\endgroup$ Commented Jan 2, 2016 at 12:19

1 Answer 1

3
$\begingroup$

Any method that uses thresholds is problematic. And note that the measures you listed do not compare classifiers; they compare predictions. Use a proper accuracy scoring rule such as Brier score or pseudo $R^2$ to get the best results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.