349
Abbreviations
AUC = Area Under the Curve.
AUROC = Area Under the Receiver Operating Characteristic curve.
AUC is used most of the time to mean AUROC, which is a bad practice since as Marc Claesen pointed out AUC is ambiguous (could be any curve) while AUROC is not.
Interpreting the AUROC
The AUROC has several equivalent interpretations:
The expectation ...
answered Jan 9 '15 at 19:15
Franck Dernoncourt
38.2k26 gold badges142 silver badges261 bronze badges
189
I'm not sure I got the question, but since the title asks for explaining ROC curves, I'll try.
ROC Curves are used to see how well your classifier can separate positive and negative examples and to identify the best threshold for separating them.
To be able to use the ROC curve, your classifier has to be ranking - that is, it should be able to rank ...
130
I would recommend Hanley’s & McNeil’s 1982 paper ‘The meaning and use of the area under a receiver operating characteristic (ROC) curve’.
Example
They have the following table of disease status and test result (corresponding to, for example, the estimated risk from a logistic model). The first number on the right is the number of patients with true ...
69
Although I'm a bit late to the party, but here's my 5 cents. @FranckDernoncourt (+1) already mentioned possible interpretations of AUC ROC, and my favorite one is the first on his list (I use different wording, but it's the same):
the AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher ...
54
Important considerations are not included in any of these discussions. The procedures discussed above invite inappropriate thresholding and utilize improper accuracy scoring rules (proportions) that are optimized by choosing the wrong features and giving them the wrong weights.
Dichotomization of continuous predictions flies in the face of optimal decision ...
53
There is quite a bit of terminological confusion in this area. Personally, I always find it useful to come back to a confusion matrix to think about this. In a classification / screening test, you can have four different situations:
Condition: A Not A
Test says “A” True positive | False positive
...
34
Thanks to all who aswered this question. I agree that there could be no one correct answer and criteria greatly depend on the aims that stand behind of the certain diagnostic test.
Finally I had found an R package OptimalCutpoints dedicated exactly to finding cutoff point in such type of analysis. Actually there are several methods of determining cutoff ...
34
Have a look at this question: Understanding ROC curve
Here's how to build a ROC curve (from that question):
Drawing ROC curve
given a data set processed by your ranking classifier
rank test examples on decreasing score
start in $(0, 0)$
for each example $x$ (in the decreasing order)
if $x$ is positive, move $1/\text{pos}$ up
if $x$ is negative, move $1/...
33
Here are the conclusions from a paper by Davis & Goadrich explaining the relationship between ROC and PR space. They answer the first two questions:
First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it ...
33
While the two statistics measures are likely to be correlated, they measure different qualities of the classifier.
AUROC
The area under the curve (AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative example. It measures the classifiers skill in ranking a set of patterns ...
32
The question is quite vague so I am going to assume you want to choose an appropriate performance measure to compare different models. For a good overview of the key differences between ROC and PR curves, you can refer to the following paper: The Relationship Between Precision-Recall and ROC Curves by Davis and Goadrich.
To quote Davis and Goadrich:
...
32
A perfect predictor gives an AUC-ROC score of 1, a predictor which makes random guesses has an AUC-ROC score of 0.5.
If you get a score of 0 that means the classifier is perfectly incorrect, it is predicting the incorrect choice 100% of the time. If you just changed the prediction of this classifier to the opposite choice then it could predict perfectly and ...
31
Improper scoring rules such as proportion classified correctly, sensitivity, and specificity are not only arbitrary (in choice of threshold) but are improper, i.e., they have the property that maximizing them leads to a bogus model, inaccurate predictions, and selecting the wrong features. It is good that they disagree with proper scoring (log-likelihood; ...
30
AUC (based on ROC) and overall accuracy seems not the same concept.
Overall accuracy is based on one specific cutpoint, while ROC tries all of the cutpoint and plots the sensitivity and specificity.
So when we compare the overall accuracy, we are comparing the accuracy based on some cutpoint. The overall accuracy varies from different cutpoint.
27
AIC and c-statistic are trying to answer different questions. (Also some issues with c-statistic have been raised in recent years, but I'll come onto that as an aside)
Roughly speaking:
AIC is telling you how good your model fits for a specific mis-classification cost.
AUC is telling you how good your model would work, on average, across all mis-...
27
The plot is ROC curve and the (False Positive Rate, True Positive Rate) points are calculated for different thresholds. Assuming you have an uniform utility function, the optimal threshold value is the one for the point closest to (0, 1).
24
There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data.
In a medical context, it's about the real world outcomes - how many people you save from dying, for example. In a medical context Sensitivity (TPR) is used to see how ...
24
The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true negatives while Dice coefficient and many other measures just handle true negatives as uninteresting defaults (see The Basics of Classifier Evaluation, Part 1).
...
23
I doubt that there is a clear and consistent distinction across statistically minded sciences and fields between regression and curve-fitting.
Regression without qualification implies linear regression and least-squares estimation. That doesn't rule out other or broader senses: indeed once you allow logit, Poisson, negative binomial regression, etc., etc. ...
23
In the general case: you can't
The ROC curve shows how sensitivity and specificity varies at every possible threshold. A contingency table has been calculated at a single threshold and information about other thresholds has been lost. Therefore you can't calculate the ROC curve from this summarized data.
But my classifier is binary, so I have one single ...
22
When you do logistic regression, you are given two classes coded as $1$ and $0$. Now, you compute probabilities that given some explanatory varialbes an individual belongs to the class coded as $1$. If you now choose a probability threshold and classify all individuals with a probability greater than this threshold as class $1$ and below as $0$, you will in ...
22
It's easy to see once you obtained a closed-form formula for AUC.
Since we have finite number of samples $\{(x_i, y_i)\}_{i=1}^N$, we'll have finite number of points on the ROC curve. We do linear interpolation in between.
First, some definitions. Suppose we'd like to evaluate an algorithm $A(x)$ that outputs a probability of $x$ lying in the positive ...
22
Let's start with an example. Say Alice is a track coach and wants to pick an athlete to represent the team in an upcoming sporting event, a 200m sprint. Naturally she wants to pick the fastest runner.
A strictly proper scoring rule would be to nominate the fastest runner of the team over the 200m distance. This maximizes exactly what coach Alice wants in ...
20
To generate ROC curves (= Receiver Operating Characteristic curves):
Assume we have a probabilistic, binary classifier such as logistic regression. Before presenting the ROC curve, the concept of confusion matrix must be understood. When we make a binary prediction, there can be 4 types of errors:
We predict 0 while we should have the class is actually 0: ...
answered Jan 19 '16 at 16:11
Franck Dernoncourt
38.2k26 gold badges142 silver badges261 bronze badges
20
These are not the same thing and they are often used in different contexts. The Dice score is often used to quantify the performance of image segmentation methods. There you annotate some ground truth region in your image and then make an automated algorithm to do it. You validate the algorithm by calculating the Dice score, which is a measure of how similar ...
20
A ROC curve visualizes TPR and FPR for all possible thresholds.
If you plot two ROC curves 'A' and 'B' and they do not cross each other, then one of your classifiers is clearly performing better, because for all possible FPR values you get a higher TPR. Obviously the area under the ROC will also be greater.
Now, if they do cross each other, then there is a ...
20
A threshold isn't trained with the model because logistic regression isn't a classifier (cf., Why isn't Logistic Regression called Logistic Classification?). It is a model to estimate the parameter, $p$, that governs the behavior of the Bernoulli distribution. That is, you are assuming that the response distribution, conditional on the covariates, is ...
answered Apr 25 '19 at 15:43
gung - Reinstate Monica
126k75 gold badges333 silver badges622 bronze badges
19
You get a nice symmetric ROC plot only when standard deviations for both outcomes are the same. If they are rather different then you may get exactly the result you describe.
The following Mathematica code demonstrates this. We assume that a target yields a normal distribution in response space, and that noise also yields a normal distribution, but a ...
19
The paper gives the following definition, which is pretty much a constructive one:
Linear interpolation is used between adjacent points.
No point lies above the final curve.
For any pair of points used to construct the curve, the line segment connecting them is equal to or below the curve.
The main problem with this one is it's not a particularly ...
19
The Gini Coefficient is the summary statistic of the Cumulative Accuracy Profile (CAP) chart. It is calculated as the quotient of the area which the CAP curve and diagonal enclose and the corresponding area in an ideal rating procedure.
Area Under Receiver Operating Characteristic curve (or AUROC for short) is the summary statistic of the ROC curve chart.
...
Only top voted, non community-wiki answers of a minimum length are eligible
