Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What other measures there are? Is precision, recall for negative sample fine or F-1 measure the best? If the model is a probability model, is AUC (Area under curve) a good measure?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Any method that uses arbitrary cutoffs and dichotomizes continuous information such as probability of class membership is problematic. And classification accuracy is an improper accuracy scoring rule, being optimized by the wrong model. Concordance probability ($c$-index; ROC area) is a measure of pure discrimination. For an overall measure consider the proper accuracy score known as the Brier score or use a generalized likelihood-based $R^2$ measure.
answered Jul 26, 2015 at 11:40
-
1$\begingroup$ Thanks!I was wondering if I use the AUC in the test set to represent the classifier's performance on the test set, or use F1 score after choosing a threshold, what the problem is? For Brier score, if the data is imbalance, and it may give very high probability(0.99..) to +1 class, and a little bigger than 0.5 probability to -1 class,then the score also will be very low, but this classifier is considerably bad. $\endgroup$ Jul 26, 2015 at 13:53
-
$\begingroup$ I think AUC is only overall performance since the threshold need to be determined after all. So the measure after threshold being determined is needed. $\endgroup$ Jul 26, 2015 at 13:55
-
4$\begingroup$ Brier score has been used for very imbalanced data since 1951. It's what the US Weather Service uses for judging the accuracy of rainfall forecasts. I don't think the precision, recall, F1 are proper scoring rules. You can supplement the Brier score with the $c$-index (concordance probability; ROC area) which require no thresholding. $\endgroup$ Jul 26, 2015 at 17:50
-
4$\begingroup$ Thanks for your answer, but why is Brier score useful for imbalanced data? Essentially, Brier score is the mean squared error of the forecast and the forecast of very rare events should have little effect on the mean, souldn't it? $\endgroup$ Jun 6, 2017 at 18:44
-
4$\begingroup$ Forecasts of rare events have the "right" effect on the mean, i.e., mean predicted probability of the event = overall proportion of events. The Brier score works no matter what the prevalence of events. For a measure of pure discrimination, the $c$-index (AUROC) has an interpretation that in fact is completely free of prevalence. $\endgroup$ Jun 7, 2017 at 12:26