I know that we assess performance of classifiers typically with metrics like accuracy, ROC, etc. typically because we want to know whether or not a classifier can accurately predict an outcome. But, what if we are more interested in the probabilistic output of a classifier? As in, the quantity we care about is not the class that is predicted but the probability of that class. How can we go about assessing the performance of the model?
-
1$\begingroup$ It is perhaps worth looking into the Brier score en.wikipedia.org/wiki/Brier_score $\endgroup$– CamMay 29, 2018 at 1:57
-
1$\begingroup$ There are metrics that measure how closely a probability aligns with a label. The most popular are the log-loss (used in logistic regression, classification trees, gradient boosting, and neural networks), and the brier score. Additionally, the AUC measures how well the probabilities put the observations in order, that is, how often a positive class is assigned a larger probability than a negative class. $\endgroup$– Matthew DruryMay 29, 2018 at 4:08
1 Answer
Look into proper scoring rules, including but not limited to the Brier score and log score. These scoring rules are used to measure the performance of probabilistic forecasts. A proper scoring rule is maximized when the predicted probabilities match the true probabilities. There also exists a taxonomy of score propriety, which includes "semi-proper" scoring rules. This Cross Validated answer provides a helpful description of the different types of score propriety.
-
1$\begingroup$ Concordance index is not a proper scoring rule, its only sensitive to ranking. $\endgroup$ May 29, 2018 at 16:44
-
1$\begingroup$ You are correct. Some would call the c-index semi-proper. I'll revise. $\endgroup$ May 29, 2018 at 21:07