Clearly, Brier Score and AUROC are better performance measures to compare classifiers. However, besides that, I am interested in a let's call it more economic view. I could imagine a classifier being much more accurate than the other. Nevertheless, the other classifier dominates in the ROC space for some sensitivity and specificity values. In introducing costs for misclassification, I might prefer exactly these values since the overall costs will be lower. I would like to point that out in the master thesis I'm writing right now. Is this correct and a legitimate point to make?
$\begingroup$
$\endgroup$
2
asked Jan 2, 2016 at 10:53
-
$\begingroup$ It is not clear to me that Brier score and AUROC are "better". I think it depends on what you are trying to do with the classifier. $\endgroup$– Peter FlomJan 2, 2016 at 12:13
-
$\begingroup$ @ Peter Flom. I guess, I wasn't so clear about that. But when I'm comparing two classifiers in general (threshold-independent), sensitivity and specificity are not well-suited since maybe for one classifier I get higher sensitivity/specificity for a threshold of 0.5 and for another one I get better values for 0.4. This is why I used AUROC and Brier Score. $\endgroup$– Patrick BaladaJan 2, 2016 at 12:19
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Any method that uses thresholds is problematic. And note that the measures you listed do not compare classifiers; they compare predictions. Use a proper accuracy scoring rule such as Brier score or pseudo $R^2$ to get the best results.
answered Jan 2, 2016 at 14:17