16
$\begingroup$

What is the advantages of the ROC curves?

For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm to select a set of features then I applied SVM for classification. In this case how can I get a ROC curve? Should I change the threshold values of my feature selection algorithm and get sensitivty and specificity of the output to draw a ROC curve?

In my case what is the purpose of creating a ROC curve?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ "An introduction to ROC analysis" by Tom Fawcett helped me to better understand ROC curves. You might enjoy it if you are looking for additional literature on the topic. $\endgroup$
    – Alexander
    Commented May 18, 2012 at 16:52
  • $\begingroup$ Okay but what would you do to compare two classifiers? If they use thresholds so that niether specificity nor sensitivity match very closely I don't think it is easy to ccompare without lookin at more poitns on the ROC. $\endgroup$
    – Michael R. Chernick
    Commented May 19, 2012 at 21:33
  • 1
    $\begingroup$ It is seldom appropriate to develop classifiers, and classification error is an improper scoring rule. There are many high-power methods for comparing quality of true predictions, and they are more intuitive than ROC curves. See for example this. $\endgroup$
    – Frank Harrell
    Commented Nov 22, 2019 at 13:42

4 Answers 4

Reset to default
14
$\begingroup$

Many binary classification algorithms compute a sort of classification score (sometimes but not always this is a probability of being in the target state), and they classify based upon whether or not the score is above a certain threshold. Viewing the ROC curve lets you see the tradeoff between sensitivity and specificity for all possible thresholds rather than just the one that was chosen by the modeling technique. Different classification objectives might make one point on the curve more suitable for one task and another more suitable for a different task, so looking at the ROC curve is a way to assess the model independent of the choice of a threshold.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Thank you for the reply. It is really useful. Based on the ROC curve is there anyway to determine the threshold? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. $\endgroup$
    – user570593
    Commented May 18, 2012 at 16:19
  • 1
    $\begingroup$ The ROC curve shows you sensitivity and specificity at all possible thresholds, so if you find a point that represents the right tradeoff, you can choose the threshold that goes with that point on the curve. $\endgroup$
    – Michael McGowan
    Commented May 18, 2012 at 16:21
  • $\begingroup$ Is there any automatic way to select the right tradeoff or should I select the tradeoff by myself? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. $\endgroup$
    – user570593
    Commented May 18, 2012 at 16:23
  • 2
    $\begingroup$ If you have a well-defined criterion (for instance maximizing precision) then this can be automated. But a good tradeoff for one problem might be lousy for another. $\endgroup$
    – Michael McGowan
    Commented May 18, 2012 at 16:25
  • 2
    $\begingroup$ Sensitivity or specificity of 100% can trivially be obtained by setting your threshold at the minimum or maximum value...is that really what you want? $\endgroup$
    – Michael McGowan
    Commented May 18, 2012 at 16:27
11
$\begingroup$

ROC curves are not informative in 99% of the cases I've seen over the past few years. They seem to be thought of as obligatory by many statisticians and even more machine learning practitioners. And make sure your problem is really a classification problem and not a risk estimation problem. At the heart of problems with ROC curves is that they invite users to use cutpoints for continuous variables, and they use backwards probabilities, i.e., probabilities of events that are in reverse time order (sensitivity and specificity). ROC curves cannot be used to find optimum tradeoffs except in very special cases where users of a decision rule abdicate their loss (cost; utility) function to the analyst.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I don't completely agree with Frank. I think using the AUC of the ROC is often a problem. But qualitatively I think it can be helpful to compare algorithms. Just looking at specificity and sensitivity at a single point is not nearly as informative. Also I am not sure that his answer really addresses the question because the OP really want to know in his case why it falls into the 99% non informative cases or the 1% that are inforamtive. $\endgroup$
    – Michael R. Chernick
    Commented May 19, 2012 at 18:22
  • 1
    $\begingroup$ @FrankHarrell But how do you compare two algorithms when they don't match very closely on either specificity and sensitivity? $\endgroup$
    – Michael R. Chernick
    Commented May 20, 2012 at 1:45
  • 7
    $\begingroup$ @Momo - I mean that viewing the ROC curve did not help understand model performance very well, and even more the ROC curve did not lead to any insight or to good behavior. Generalized $R^2$ measures are simple translations of likelihood ratio $\chi^2$ statistics and so are powerful. Michael - I'm not really interested in sensitivity or specificity because of the backwards time ordering. I want to know that a predicted risk of 0.2 means that the actual risk is very close to 0.2, and I want decent predictive discrimination as measured by a rank correlation measure or generalized $R^2$. $\endgroup$
    – Frank Harrell
    Commented May 20, 2012 at 3:32
  • 1
    $\begingroup$ Thanks, @FrankHarrell. I'm still not sure I completely follow the arguments against the usage of ROC curve and sens spec. Maybe it's time to ask a question myself. $\endgroup$
    – Momo
    Commented May 20, 2012 at 11:21
  • 1
    $\begingroup$ @Momo I might be able to provide more info if you state the ultimate goal of your analysis as well as how you think an ROC curve advances that goal. Be sure to include what you do when a prediction is equivocal. $\endgroup$
    – Frank Harrell
    Commented May 20, 2012 at 12:20
1
$\begingroup$

After creating a ROC curve, the AUC (area under the curve) can be calculated. The AUC is accuracy of the test across many thresholds. AUC = 1 means the test is perfect. AUC = .5 means performs at chance for binary classification.

If there are multiple models, AUC provides a single measurement to compare across different models. There are always trade-offs with any single measure but AUC is a good place to start.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I think this is hazardous thinking. The AUROC is useful IMHO only because in the binary $Y$ case it equals the concordance probability ($c$-index; Wilcoxon statistic; Somers' $D_{xy}$ rank correlation coefficient). Drawing the entire ROC leads to dichotomous thinking. $\endgroup$
    – Frank Harrell
    Commented May 22, 2014 at 14:58
1
$\begingroup$

The AUC does not compare classes real vs. predicted with each other. It is not looking at the predicted class, but the prediction score or the probability. You can do the prediction of the class by applying a cutoff to this score, say, every sample that got a score below 0.5 is classified as negative. But the ROC comes before that happens. It is working with the scores/class-probabilities.

It takes these scores and sorts all samples according to that score. Now, whenever you find a positive sample the ROC-curve makes a step up (along the y-axis). Whenever you find a negative sample you move right (along the x-axis). If that score is different for the two classes, the positive samples come first (usually). That means you make more steps up than to the right. Further down the list the negative samples will come, so you move left. When you are through the whole list of samples you reach at the coordinate (1,1) which corresponds to 100% of the positive and 100% of the negative samples.

If the score perfectly separates the positive from the negative samples you move all the way from (x=0, y=0) to (1,0) and then from there to (1, 1). So, the area under the curve is 1.

If your score has the same distribution for positive and negative samples the probabilities to find a positive or negative sample in the sorted list are equal and therefore the probabilities to move up or left in the ROC-curve are equal. That is why you move along the diagonal, because you essentially move up and left, and up and left, and so on... which gives an AROC value of around 0.5.

In the case of an imbalanced dataset, the stepsize is different. So, you make smaller steps to the left (if you have more negative samples). That is why the score is more or less independent of the imbalance.

So with the ROC curve, you can visualize how your samples are separated and the area under the curve can be a very good metric to measure the performance of a binary classification algorithm or any variable that may be used to separate classes.

enter image description here

The figure shows the same distributions with different sample sizes. The black area shows where ROC-curves of random mixtures of positive and negative samples would be expected.

Improve this answer
$\endgroup$
4
  • $\begingroup$ These graphs provide no insight and have an exceptionally high ink:information ratio IMHO. Stick with proper accuracy scores: fharrell.com/post/class-damage fharrell.com/post/addvalue $\endgroup$
    – Frank Harrell
    Commented Nov 16, 2019 at 13:21
  • $\begingroup$ There is way more information in these graphs than in a single one dimensional accuracy score. The same score can come from many different distributions. Do you have early recognition? Do you have multiple classes of positive samples that behave differently? Is your result statistically significant? All those questions can be obvious to answer by looking at those graphs and impossible to address with a single accuracy score. $\endgroup$
    – user209249
    Commented Nov 16, 2019 at 22:56
  • $\begingroup$ I seriously question that consumers and analysts can get insight from these curves that is anywhere near as intuitive as showing a calibration curve overlaid with a high-resolution histogram showing the predicted values. And each point on the ROC curve is an improper accuracy score. $\endgroup$
    – Frank Harrell
    Commented Nov 21, 2019 at 2:03
  • $\begingroup$ Beginners often have a hard time understanding these curves. Therefore, I wouldn't necessarily recommend to show it to consumers in order to advertise your product. I think, there you want something that is more simplistic. The curve is more than the individual points though. $\endgroup$
    – user209249
    Commented Nov 21, 2019 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.