48
votes
Can AUC-ROC be between 0-0.5?
I am sorry, but these answers are dangerously wrong. No, you cannot just flip AUC after you see the data. Imagine you are buying stocks, and you always bought the wrong one, but you said to yourself, ...
- 7,789
40
votes
Accepted
Can AUC-ROC be between 0-0.5?
A perfect predictor gives an AUC-ROC score of 1, a predictor which makes random guesses has an AUC-ROC score of 0.5.
If you get a score of 0 that means the classifier is perfectly incorrect, it is ...
- 4,159
37
votes
What does it mean that AUC is a semi-proper scoring rule?
Let's start with an example. Say Alice is a track coach and wants to pick an athlete to represent the team in an upcoming sporting event, a 200m sprint. Naturally she wants to pick the fastest runner.
...
- 46.1k
36
votes
ROC vs precision-and-recall curves
There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data.
In a ...
- 1,500
35
votes
ROC vs Precision-recall curves on imbalanced dataset
First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare ...
- 3,859
35
votes
Accepted
What is the formula to calculate the area under the ROC curve from a contingency table?
In the general case: you can't
The ROC curve shows how sensitivity and specificity varies at every possible threshold. A contingency table has been calculated at a single threshold and information ...
- 3,859
34
votes
Is the Dice coefficient the same as accuracy?
The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
- 461
23
votes
Where in the ROC curve does it tell you what the threshold is?
Each (FPR, TPR) point on a ROC curve is associated with a threshold. However, the thresholds are not typically drawn on the curve itself.
It is possible to reveal them, either adding extra annotation ...
- 3,859
22
votes
What is the difference between GINI and AUC curve interpretation?
The Gini Coefficient is the summary statistic of the Cumulative Accuracy Profile (CAP) chart. It is calculated as the quotient of the area which the CAP curve and diagonal enclose and the ...
- 371
22
votes
How to derive the probabilistic interpretation of the AUC?
From Measuring classifier performance: a coherent alternative to the area under the ROC curve :
First thing, let's try to define the area under the ROC curve
formally. Some assumptions and ...
- 429
22
votes
Accepted
Comparison of two models when the ROC curves cross each other
A ROC curve visualizes TPR and FPR for all possible thresholds.
If you plot two ROC curves 'A' and 'B' and they do not cross each other, then one of your classifiers is clearly performing better, ...
- 2,051
21
votes
Accepted
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
A threshold isn't trained with the model because logistic regression isn't a classifier (cf., Why isn't Logistic Regression called Logistic Classification?). It is a model to estimate the parameter, $...
20
votes
What does AUC stand for and what is it?
The answers in this forum are great and I come back here often for reference. However, one thing was always missing. From @Frank's answer, we see the interpretation of AUC as the probability that a ...
- 2,630
20
votes
How to derive the probabilistic interpretation of the AUC?
@alebu's answer is great. But its notation is nonstandard and uses 0 for the positive class and 1 for the negative class. Below are the results for the standard notation (0 for the negative class and ...
- 976
19
votes
Accepted
AUPRC vs. AUC-ROC?
ROC AUC is the area under the curve where x is false positive rate (FPR) and y is true positive rate (TPR).
PR AUC is the area under the curve where x is recall and y is precision.
recall = TPR = ...
- 5,800
19
votes
Determine how good an AUC is (Area under the Curve of ROC)
From the comments:
Calimo: If you are a trader and you can get an AUC of 0.501 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your ...
19
votes
Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?
This is actually a very simple issue. The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an ...
- 98.5k
18
votes
How to determine the optimal threshold for a classifier and generate ROC curve?
The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false ...
- 728
18
votes
Accepted
Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
Wojtek J. Krzanowski and David J. Hand ROC Curves for Continuous Data (2009) is a great reference for all things related to ROC curves. It collects together a number of results in what is a ...
- 94.1k
17
votes
How to determine the optimal threshold for a classifier and generate ROC curve?
Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.
- 279
17
votes
Accepted
AUC and class imbalance in training/test dataset
It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will ...
- 3,219
17
votes
ROC vs Precision-recall curves on imbalanced dataset
Your example is definitely correct.
However, I think in the context of Kaggle competition / real life application, a skewed dataset usually means a dataset with much less positive samples than ...
- 171
17
votes
Can AUC-ROC be between 0-0.5?
They can, if the system you're analyzing performs below chance level. Trivially, you could easily construct a classifier with 0 AUC by having it always answer opposite to the truth.
In practice of ...
- 7,762
17
votes
Accepted
What is the origin of the "receiver operating characteristic" (ROC) terminology?
The earliest book reference that I know of is
Woodward, P. M. (1953). Probability and information theory with applications to radar. London: Pergamon Press.
but the concept, which was developed ...
- 47.8k
17
votes
Accepted
Dichotomizing continuous variables at their optimal cut-off for clinical interpretation
Dichotomizing a continuous covariate is ill-advised, as has been noted by other users.
One strategy I employ is to rescale the predictor to something more reasonable. 1 mmHg may not be a very ...
- 39.6k
16
votes
ROC vs precision-and-recall curves
TL;DR
$AUC_{PvR}$ highlights the amount of False Positives relative to the class size, whereas $AUC_{ROC}$ better reflects the total amount of False Positives independent of in which class they come ...
- 299
16
votes
Is up- or down-sampling imbalanced data actually that effective? Why?
The short answer appears to be Yes: there is some evidence that upsampling of the minority class and/or downsampling of the majority class in a training set can somewhat improve out-of-sample AUC (...
- 12.9k
15
votes
AUC and class imbalance in training/test dataset
(a 3-years late answer, but maybe still useful!)
ROC is sensitive to the class-imbalance issue, meaning that it favors the class with larger population solely because of its higher population. In ...
- 383
15
votes
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
It's because the optimal threshold is not only a function of the true positive rate (TPR), the false positive rate (FPR), accuracy or whatever else. The other crucial ingredient is the cost and the ...
- 131k
14
votes
How to choose between ROC AUC and F1 score?
Above answers are both good.
But what I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $Skew=\frac{negative\;examples}{positive\;...
- 401
Only top scored, non community-wiki answers of a minimum length are eligible