Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What other measures there are? Is precision, recall for negative sample fine or F-1 measure the best? If the model is a probability model, is AUC (Area under curve) a good measure?


Any method that uses arbitrary cutoffs and dichotomizes continuous information such as probability of class membership is problematic. And classification accuracy is an improper accuracy scoring rule, being optimized by the wrong model. Concordance probability ($c$-index; ROC area) is a measure of pure discrimination. For an overall measure consider the proper accuracy score known as the Brier score or use a generalized likelihood-based $R^2$ measure.

  • 1
    $\begingroup$ Thanks!I was wondering if I use the AUC in the test set to represent the classifier's performance on the test set, or use F1 score after choosing a threshold, what the problem is? For Brier score, if the data is imbalance, and it may give very high probability(0.99..) to +1 class, and a little bigger than 0.5 probability to -1 class,then the score also will be very low, but this classifier is considerably bad. $\endgroup$ – user83176 Jul 26 '15 at 13:53
  • $\begingroup$ I think AUC is only overall performance since the threshold need to be determined after all. So the measure after threshold being determined is needed. $\endgroup$ – user83176 Jul 26 '15 at 13:55
  • 3
    $\begingroup$ Brier score has been used for very imbalanced data since 1951. It's what the US Weather Service uses for judging the accuracy of rainfall forecasts. I don't think the precision, recall, F1 are proper scoring rules. You can supplement the Brier score with the $c$-index (concordance probability; ROC area) which require no thresholding. $\endgroup$ – Frank Harrell Jul 26 '15 at 17:50
  • 2
    $\begingroup$ Thanks for your answer, but why is Brier score useful for imbalanced data? Essentially, Brier score is the mean squared error of the forecast and the forecast of very rare events should have little effect on the mean, souldn't it? $\endgroup$ – Funkwecker Jun 6 '17 at 18:44
  • 3
    $\begingroup$ Forecasts of rare events have the "right" effect on the mean, i.e., mean predicted probability of the event = overall proportion of events. The Brier score works no matter what the prevalence of events. For a measure of pure discrimination, the $c$-index (AUROC) has an interpretation that in fact is completely free of prevalence. $\endgroup$ – Frank Harrell Jun 7 '17 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.