How to choose between ROC AUC and F1 score? I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?
Btw, I read the article here What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.
Thanks in advance for any help!
 A: To put in very simple words when you have a data imbalance i.e., the difference between the number of examples you have for positive and negative classes is large, you should always use F1-score. Otherwise you can use ROC/AUC curves.
A: Despite the less interpretable graph that AUC integrates, the number itself tells you the probability that a randomly chosen positive would be ranked higher than a randomly chosen negative. This is a nice summary of the degree to which positive examples are scored higher than negative examples. If the negatives are ranked higher than all the positives, your AUC is 0. If your negatives are ranked lower than all the positives, the AUC is 1. If the negatives are in the middle or scattered randomly, AUC is around 0.5. Every time your model performance degrades to the point that a positive and negative instance trade ranks when sorted by model score, AUC decreases by a constant number equal to 1/(number of positives x number of negatives).
If you have one negative and 99 positive examples, and that one negative example is ranked higher than all the positive examples, ROC AUC is 0 but you can still achieve a high F1. With a threshold at or lower than your lowest model score (0.5 will work if your model scores everything higher than 0.5), precision and recall are 99% and 100% respectively, leaving your F1 ~99.5%.
In this example, your model performed far worse than a random number generator since it assigned its highest confidence to the only negative example in the dataset. At the same time, it may well be very successful if you care about precision and recall--the problem was so easy even a random number generator could do it!
As a rule of thumb, I've found AUC is useful for comparing models as you're experimenting since it will tell you if you have a bad model despite an easy problem. Precision, recall, F1, and anything that relies on thresholds are useful once you're trying to figure out whether and to what extent it would meet production requirements.
A: If the objective of classification is scoring by probability, it is better to use AUC which averages over all possible thresholds. However, if the objective of classification just needs to classify between two possible classes and doesn't require how likely each class is predicted by the model, it is more appropriate to rely on F-score using a particular threshold.
A: Calculation formula：


*

*Precision TP/(TP+FP)

*Recall: TP/(TP+FN)

*F1-score： 2/(1/P+1/R) 

*ROC/AUC： TPR=TP/(TP+FN), FPR=FP/(FP+TN)


ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.
Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not ROC/AUC.
So in the real world, the PR curve is used more since positive and negative samples are very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.
If you just do the experiment in research papers, you can use the ROC, the experimental results will be more beautiful. On another hand, PR curve use in the real problem, and it has better interpretability. 
A: None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model.  Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$.  The $c$-index (AUROC; concordance probability) is not proper but is good for describing a single model.  It is not sensitive enough to use for choosing models or comparing even as few as two models.
A: Above answers are both good. 
But what I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $Skew=\frac{negative\;examples}{positive\;examples}$ is large). This kind of situations is very common in action detection, fraud detection, bankruptcy prediction ect. That is, the positive examples you care have relatively low rates of occurrence. 
With imbalanced data, the AUC still gives you specious value around 0.8. However, it is high due to large FP, rather than the large TP (True positive).
Such as the example below, 
TP=155,   FN=182
FP=84049, TN=34088

So when you use AUC to measure the performance of classifier, the problem is the increasing of AUC doesn't really reflect a better classifier. It's just the side-effect of too many negative examples. You can simply try in you imbalanced dataset, you will see this issue. 
The paper Facing Imbalanced Data Recommendations for the Use of Performance Metrics found "while ROC was unaffected by skew, the precision-recall curves suggest that ROC may mask poor performance in some cases." Searching for a good performance metrics is still a open question. A general F1-score may help $$
F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$
where the $\beta$ is the relative importance of precision comparing to recall. 
Then, my suggestions for imbalanced data are similar to this post. You can also try the decile table, which can be construct by searching "Two-by-Two Classification and Decile Tables". Meanwhile, I am also studying on this problem and will give better measure. 
A: For some multi class classification problems, analyzing and visualizing ROC/AUC is not straightforward. You may look into this question, How to plot ROC curves in multiclass classification?. Under such situation, using F1 score could be a better metric. 
And F1 score is a common choice for information retrieval problem and popular in industry settings. Here is an well explained example, Building ML models is hard. Deploying them in real business environments is harder.
A: Lets start with some formula to see how each measure is calculated (see Wikipedia for a complete list)：

*

*Precision: $\frac{TP}{TP+FP}$

*Recall: $\frac{TP}{TP+FN}$

*F1-score： $\frac{2}{\frac{1}{Precision}+\frac{1}{Recall}}=2\times\frac{Precision \times Recall}{Precision + Recall}$

*AUC curve is built using the following measures:

*

*TPR = $\frac{TP}{TP+FN}$=Recall

*FPR = $\frac{FP}{FP+TN}$



*PR curve is built using the following measures：

*

*Precision

*Recall
Notice that the AUC is using TPR and FPR criteria. In contrast, the the PR (Precision-Recall) curve and F1-score are using Precision, Recall criteria.
Note that the Recall = TPR used in both measures and is identical. So we only focus on the Precision and FPR to see the difference between them.
In general both measures will nicely assess the performance of a classifier.
Their difference is pronounced when classes are imbalanced, i.e. when number of samples in the positive class (say class Rare) is very small compared to the negative class (say class Freq).
When a classifier is (wrongly) predicting many samples from Freq class as Rare class, then the Precision is going to be small, but FPR could still be large (see the equation above). As a result, the PR curve will more drastically show this lack of performance compared to AUC.
The real-world data tend to be imbalance and often the Rare class is of interest. In such a cases, the using PR curve is recommended. The F1-score produces a single number which is more convenient to work with. So when many classifiers are being compared (or during hyper parameter optimisation), then F1-score is used instead of drawing a PR curve.
