48
votes
Can AUC-ROC be between 0-0.5?
I am sorry, but these answers are dangerously wrong. No, you cannot just flip AUC after you see the data. Imagine you are buying stocks, and you always bought the wrong one, but you said to yourself, ...
- 7,789
40
votes
Accepted
Can AUC-ROC be between 0-0.5?
A perfect predictor gives an AUC-ROC score of 1, a predictor which makes random guesses has an AUC-ROC score of 0.5.
If you get a score of 0 that means the classifier is perfectly incorrect, it is ...
- 4,159
35
votes
Accepted
What is the formula to calculate the area under the ROC curve from a contingency table?
In the general case: you can't
The ROC curve shows how sensitivity and specificity varies at every possible threshold. A contingency table has been calculated at a single threshold and information ...
- 3,859
34
votes
Is the Dice coefficient the same as accuracy?
The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
- 461
34
votes
Accepted
Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?
This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on.
This question can almost certainly not be answered well for you by readers at ...
- 20.4k
23
votes
Where in the ROC curve does it tell you what the threshold is?
Each (FPR, TPR) point on a ROC curve is associated with a threshold. However, the thresholds are not typically drawn on the curve itself.
It is possible to reveal them, either adding extra annotation ...
- 3,859
22
votes
How to derive the probabilistic interpretation of the AUC?
From Measuring classifier performance: a coherent alternative to the area under the ROC curve :
First thing, let's try to define the area under the ROC curve
formally. Some assumptions and ...
- 429
22
votes
Accepted
Comparison of two models when the ROC curves cross each other
A ROC curve visualizes TPR and FPR for all possible thresholds.
If you plot two ROC curves 'A' and 'B' and they do not cross each other, then one of your classifiers is clearly performing better, ...
- 2,051
20
votes
What does AUC stand for and what is it?
The answers in this forum are great and I come back here often for reference. However, one thing was always missing. From @Frank's answer, we see the interpretation of AUC as the probability that a ...
- 2,630
20
votes
How to derive the probabilistic interpretation of the AUC?
@alebu's answer is great. But its notation is nonstandard and uses 0 for the positive class and 1 for the negative class. Below are the results for the standard notation (0 for the negative class and ...
- 976
19
votes
Accepted
AUPRC vs. AUC-ROC?
ROC AUC is the area under the curve where x is false positive rate (FPR) and y is true positive rate (TPR).
PR AUC is the area under the curve where x is recall and y is precision.
recall = TPR = ...
- 5,800
19
votes
Determine how good an AUC is (Area under the Curve of ROC)
From the comments:
Calimo: If you are a trader and you can get an AUC of 0.501 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your ...
19
votes
Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?
This is actually a very simple issue. The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an ...
- 98.5k
18
votes
Accepted
Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
Wojtek J. Krzanowski and David J. Hand ROC Curves for Continuous Data (2009) is a great reference for all things related to ROC curves. It collects together a number of results in what is a ...
- 94.1k
17
votes
Accepted
AUC and class imbalance in training/test dataset
It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will ...
- 3,219
17
votes
Can AUC-ROC be between 0-0.5?
They can, if the system you're analyzing performs below chance level. Trivially, you could easily construct a classifier with 0 AUC by having it always answer opposite to the truth.
In practice of ...
- 7,762
16
votes
Accepted
logloss vs gini/auc
Whereas the AUC is computed with regards to binary classification with a varying decision threshold, logloss actually takes "certainty" of classification into account.
Therefore to my understanding, ...
- 3,491
15
votes
Accepted
optimizing auc vs logloss in binary classification problems
As you mention, AUC is a rank statistic (i.e. scale invariant) & log loss is a calibration statistic. One may trivially construct a model which has the same AUC but fails to minimize log loss w.r....
- 797
15
votes
AUC and class imbalance in training/test dataset
(a 3-years late answer, but maybe still useful!)
ROC is sensitive to the class-imbalance issue, meaning that it favors the class with larger population solely because of its higher population. In ...
- 383
15
votes
Does a logistic regression maximizing likelihood necessarily also maximize AUC over linear models?
It is not the case that $\beta_{MLE} \equiv \beta_{AUC}$.
To illustrate this, consider that AUC can written as
$P(\hat y_1 > \hat y_0 | y_1 = 1, y_0 = 0)$
In otherwords, the ordering of the ...
- 21.6k
15
votes
Accepted
Logarithmic loss vs Brier score vs AUC score
The choice depends on how you plan to use the model. There are many potential strictly proper scoring rules (AUC isn't one). They effectively put different weights on different parts of the ...
- 102k
14
votes
AUC and class imbalance in training/test dataset
I think it is not safe to say that the AUC is insensitive to class imbalance, as it introduces some confusion to the reader. In case you mean that the score itself doesn't detect class imbalance, that'...
- 241
14
votes
Determine how good an AUC is (Area under the Curve of ROC)
This is a complementary to Andrey's answer (+1).
When looking for a generally accepted reference on AUC-ROC values, I came across Hosmer's "Applied Logistic Regression". In Chapt. 5 "...
- 46.1k
14
votes
Accepted
Is higher AUC always better?
AUC is a simplified performance measure
AUC collapses the ROC curve into a single number. Because of that a comparison of two ROC curves based on AUC might miss out on particular details that are left ...
- 86.6k
13
votes
How can we calculate ROC AUC for classification algorithm such as random forest?
Although the randomForest package does not have a built-in function to generate a ROC curve and an AUC measure, it is very easy to generate in a case of 2 classes by using it in combination with the ...
- 150
13
votes
When is an AUC score misleadingly high?
One possible reason you can get high AUROC with what some might consider a mediocre prediction is if you have imbalanced data (in favor of the "zero" prediction), high recall, and low ...
- 3,454
13
votes
Accepted
Sklearn Average_Precision_Score vs. AUC
AUC (or AUROC, area under receiver operating characteristic) and AUPR (area under precision recall curve) are threshold-independent methods for evaluating a threshold-based classifier (i.e. logistic ...
- 1,923
12
votes
ROC curve for discrete classifiers like SVM: Why do we still call it a "curve"?, Isn't it just a "point"?
Normally, the predicted label $\hat{y}$ from SVM is given by
$\hat{y}=\mbox{sign}({\mathbf w^T x}+b)$, where ${\mathbf w}$ is the SVM-optimized weights of the hyper-plane, and the $b$ is the SVM-...
- 121
12
votes
Accepted
How to improve F1 score with skewed classes?
Most of the classification problems I've tackled are similar in nature, so a large class imbalance is quite common.
It is not clear whether you are using training-validation sets to build and fine ...
- 1,708
12
votes
Accepted
Generate synthetic data given AUC
There are multiple ways to do it. One is to assume to transform AUC to cohen's D and then just sample data from 2 standard normal distributions D standard deviations apart.
We can transform AUC to D ...
- 7,789
Only top scored, non community-wiki answers of a minimum length are eligible