Timeline for What's the measure to assess the binary classification accuracy for imbalanced data?
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 7, 2017 at 12:26 | comment | added | Frank Harrell | Forecasts of rare events have the "right" effect on the mean, i.e., mean predicted probability of the event = overall proportion of events. The Brier score works no matter what the prevalence of events. For a measure of pure discrimination, the $c$-index (AUROC) has an interpretation that in fact is completely free of prevalence. | |
| Jun 6, 2017 at 18:44 | comment | added | Funkwecker | Thanks for your answer, but why is Brier score useful for imbalanced data? Essentially, Brier score is the mean squared error of the forecast and the forecast of very rare events should have little effect on the mean, souldn't it? | |
| Jul 26, 2015 at 17:50 | comment | added | Frank Harrell | Brier score has been used for very imbalanced data since 1951. It's what the US Weather Service uses for judging the accuracy of rainfall forecasts. I don't think the precision, recall, F1 are proper scoring rules. You can supplement the Brier score with the $c$-index (concordance probability; ROC area) which require no thresholding. | |
| Jul 26, 2015 at 17:10 | comment | added | user83176 | I see. Because we finally use a Logistic Regression, and compute its predicted class and find the precision,recall,F1 and something. I will see if Brier score can give a very good measure for imbalance data. | |
| Jul 26, 2015 at 16:21 | comment | added | Frank Harrell | The Brief score does not use thresholds in any way. It's interpretation will vary a bit depending on prevalence of $Y=1$. Don't choose a threshold at any rate, unless you possess the utility function. There is no need to choose a threshold in most cases. Don't use a measure that uses a threshold as this will be very arbitrary and imprecise and often represents an improper accuracy scoring rule. | |
| Jul 26, 2015 at 13:55 | comment | added | user83176 | I think AUC is only overall performance since the threshold need to be determined after all. So the measure after threshold being determined is needed. | |
| Jul 26, 2015 at 13:53 | comment | added | user83176 | Thanks!I was wondering if I use the AUC in the test set to represent the classifier's performance on the test set, or use F1 score after choosing a threshold, what the problem is? For Brier score, if the data is imbalance, and it may give very high probability(0.99..) to +1 class, and a little bigger than 0.5 probability to -1 class,then the score also will be very low, but this classifier is considerably bad. | |
| Jul 26, 2015 at 11:40 | history | answered | Frank Harrell | CC BY-SA 3.0 |