I recently completed a Kaggle competition in which ROC AUC score was used as per competition requirement. Before this project, I normally used F1-score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?
I read What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.
None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model. Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$. The $c$-index (AUROC; concordance probability) is not proper but is good for describing a single model. It is not sensitive enough to use for choosing models or comparing even as few as two models.
Calculation formula：
ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.
Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not on ROC/AUC.
So in the real world, the PR curve is used more since it is common for positive and negative samples to be very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.
Use the ROC if you want to report experimental results in research papers, because the experimental results will be more beautiful. On the other hand, use the PR curve in a real-world problem as it has better interpretability.
Above answers are both good.
But what I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $Skew=\frac{negative\;examples}{positive\;examples}$ is large). This kind of situations is very common in action detection, fraud detection, bankruptcy prediction ect. That is, the positive examples you care have relatively low rates of occurrence.
With imbalanced data, the AUC still gives you suspicious value around 0.8. However, it is high due to large TN, rather than the large TP (True positive).
Such as the example below,
TP=155, FN=182
FP=84049, TN=34088
So when you use AUC to measure the performance of classifier, the problem is the increasing of AUC doesn't really reflect a better classifier. It's just the side-effect of too many negative examples. You can simply try in you imbalanced dataset, you will see this issue.
The paper Facing Imbalanced Data Recommendations for the Use of Performance Metrics found "while ROC was unaffected by skew, the precision-recall curves suggest that ROC may mask poor performance in some cases." Searching for a good performance metrics is still a open question. A general F1-score may help $$ F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$
where the $\beta$ is the relative importance of precision comparing to recall.
Then, my suggestions for imbalanced data are similar to this post. You can also try the decile table, which can be construct by searching "Two-by-Two Classification and Decile Tables". Meanwhile, I am also studying on this problem and will give better measure.
Years later, we developed a new goodness-of-fit measure to resemble the OLS R-squared that can imply the proportion of the variance of the surrogate response explained by explanatory variables. We have had the paper "A new goodness-of-fit measure for probit models: surrogate 𝑅^2" published with all the details. Please check the package webpage as well: https://xiaorui.site/SurrogateRsq/.
To put in very simple words when you have a data imbalance i.e., the difference between the number of examples you have for positive and negative classes is large, you should always use F1-score. Otherwise you can use ROC/AUC curves.
Despite the less interpretable graph that AUC integrates, the number itself tells you the probability that a randomly chosen positive would be ranked higher than a randomly chosen negative. This is a nice summary of the degree to which positive examples are scored higher than negative examples. If the negatives are ranked higher than all the positives, your AUC is 0. If your negatives are ranked lower than all the positives, the AUC is 1. If the negatives are in the middle or scattered randomly, AUC is around 0.5. Every time your model performance degrades to the point that a positive and negative instance trade ranks when sorted by model score, AUC decreases by a constant number equal to 1/(number of positives x number of negatives).
If you have one negative and 99 positive examples, and that one negative example is ranked higher than all the positive examples, ROC AUC is 0 but you can still achieve a high F1. With a threshold at or lower than your lowest model score (0.5 will work if your model scores everything higher than 0.5), precision and recall are 99% and 100% respectively, leaving your F1 ~99.5%.
In this example, your model performed far worse than a random number generator since it assigned its highest confidence to the only negative example in the dataset. At the same time, it may well be very successful if you care about precision and recall--the problem was so easy even a random number generator could do it!
As a rule of thumb, I've found AUC is useful for comparing models as you're experimenting since it will tell you if you have a bad model despite an easy problem. Precision, recall, F1, and anything that relies on thresholds are useful once you're trying to figure out whether and to what extent it would meet production requirements.
If the objective of classification is scoring by probability, it is better to use AUC which averages over all possible thresholds. However, if the objective of classification just needs to classify between two possible classes and doesn't require how likely each class is predicted by the model, it is more appropriate to rely on F-score using a particular threshold.
Lets start with some formula to see how each measure is calculated (see Wikipedia for a complete list)：
Precision: $\frac{TP}{TP+FP}$Recall: $\frac{TP}{TP+FN}$F1-score： $\frac{2}{\frac{1}{Precision}+\frac{1}{Recall}}=2\times\frac{Precision \times Recall}{Precision + Recall}$AUC curve is built using the following measures:
TPR = $\frac{TP}{TP+FN}$=RecallFPR = $\frac{FP}{FP+TN}$PR curve is built using the following measures：
PrecisionRecallNotice that the AUC is using TPR and FPR criteria. In contrast, the the PR (Precision-Recall) curve and F1-score are using Precision, Recall criteria.
Note that the Recall = TPR used in both measures and is identical. So we only focus on the Precision and FPR to see the difference between them.
In general both measures will nicely assess the performance of a classifier.
Their difference is pronounced when classes are imbalanced, i.e. when number of samples in the positive class (say class Rare) is very small compared to the negative class (say class Freq).
When a classifier is (wrongly) predicting many samples from Freq class as Rare class, then the Precision is going to be small, but FPR could still be large (see the equation above). As a result, the PR curve will more drastically show this lack of performance compared to AUC.
The real-world data tend to be imbalance and often the Rare class is of interest. In such a cases, the using PR curve is recommended. The F1-score produces a single number which is more convenient to work with. So when many classifiers are being compared (or during hyper parameter optimisation), then F1-score is used instead of drawing a PR curve.
The real question is always "how well are the business goals met?". Sometimes that aligns with a standard metric, but not often.
If you are screening drug candidates, and have budget for trials with 10, then what really matters is the probability that there is an excellent candidate among the top ten - which is different to ROC AUC, PR AUC, or F1.
If you are screening patients for cancer, then false negatives are very bad - but you don't want an excessive number of false positives either, because of the related cost and expense. It is always context dependent.
For some multi class classification problems, analyzing and visualizing ROC/AUC is not straightforward. You may look into this question, How to plot ROC curves in multiclass classification?. Under such situation, using F1 score could be a better metric.
And F1 score is a common choice for information retrieval problem and popular in industry settings. Here is an well explained example, Building ML models is hard. Deploying them in real business environments is harder.
Single number metrics are nice for Kaggle. In real life, as said above, you need to solve a business need, and the metric needs to be derived from that.
I rarely see cases where the CEO comes to the developers and says "well folks, we really need to improve the harmonic mean of precision and recall". We also rarely need to optimize the ROC AUC. What really gives us insight is the plot of ROC curve and P-R curve. My 2 cents.
