Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 11 down vote favorite
2

What is the advantages of the ROC curves?

For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm to select a set of features then I applied SVM for classification. In this case how can I get a ROC curve? Should I change the threshold values of my feature selection algorithm and get sensitivty and specificity of the output to draw a ROC curve?

In my case what is the purpose of creating a ROC curve?

share|improve this question
1  
"An introduction to ROC analysis" by Tom Fawcett helped me to better understand ROC curves. You might enjoy it if you are looking for additional literature on the topic. – Alexander May 18 '12 at 16:52
Okay but what would you do to compare two classifiers? If they use thresholds so that niether specificity nor sensitivity match very closely I don't think it is easy to ccompare without lookin at more poitns on the ROC. – Michael Chernick May 19 '12 at 21:33

migrated from stackoverflow.com May 18 '12 at 16:00

2 Answers

up vote 6 down vote accepted

Many binary classification algorithms compute a sort of classification score (sometimes but not always this is a probability of being in the target state), and they classify based upon whether or not the score is above a certain threshold. Viewing the ROC curve lets you see the tradeoff between sensitivity and specificity for all possible thresholds rather than just the one that was chosen by the modeling technique. Different classification objectives might make one point on the curve more suitable for one task and another more suitable for a different task, so looking at the ROC curve is a way to assess the model independent of the choice of a threshold.

share|improve this answer
Thank you for the reply. It is really useful. Based on the ROC curve is there anyway to determine the threshold? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. – user570593 May 18 '12 at 16:19
The ROC curve shows you sensitivity and specificity at all possible thresholds, so if you find a point that represents the right tradeoff, you can choose the threshold that goes with that point on the curve. – Michael McGowan May 18 '12 at 16:21
Is there any automatic way to select the right tradeoff or should I select the tradeoff by myself? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. – user570593 May 18 '12 at 16:23
1  
If you have a well-defined criterion (for instance maximizing precision) then this can be automated. But a good tradeoff for one problem might be lousy for another. – Michael McGowan May 18 '12 at 16:25
1  
Sensitivity or specificity of 100% can trivially be obtained by setting your threshold at the minimum or maximum value...is that really what you want? – Michael McGowan May 18 '12 at 16:27
show 7 more comments
up vote 4 down vote

ROC curves are not informative in 99% of the cases I've seen over the past few years. They seem to be thought of as obligatory by many statisticians and even more machine learning practitioners. And make sure your problem is really a classification problem and not a risk estimation problem. At the heart of problems with ROC curves is that they invite users to use cutpoints for continuous variables, and they use backwards probabilities, i.e., probabilities of events that are in reverse time order (sensitivity and specificity). ROC curves cannot be used to find optimum tradeoffs except in very special cases where users of a decision rule abdicate their loss (cost; utility) function to the analyst.

share|improve this answer
I don't completely agree with Frank. I think using the AUC of the ROC is often a problem. But qualitatively I think it can be helpful to compare algorithms. Just looking at specificity and sensitivity at a single point is not nearly as informative. Also I am not sure that his answer really addresses the question because the OP really want to know in his case why it falls into the 99% non informative cases or the 1% that are inforamtive. – Michael Chernick May 19 '12 at 18:22
Hi Michael - I'll just add that I like to use proper scoring rules including generalized $R^2$ measures, Brier score (like mean squared error). What I really like to do is to show that predictions are accurate. – Frank Harrell May 19 '12 at 21:11
@FrankHarrell I'd be interested in what you mean by "99% were not informative"? What type of cases were they and when would using the ROCC be correct? Also how are generalized $R^2$ measures better for binary problems? I am curious because it seems you have quite a strong sentiment against ROCC there and I wonder what's the background. – Momo May 19 '12 at 23:54
@FrankHarrell But how do you compare two algorithms when they don't match very closely on either specificity and sensitivity? – Michael Chernick May 20 '12 at 1:45
4  
@Momo - I mean that viewing the ROC curve did not help understand model performance very well, and even more the ROC curve did not lead to any insight or to good behavior. Generalized $R^2$ measures are simple translations of likelihood ratio $\chi^2$ statistics and so are powerful. Michael - I'm not really interested in sensitivity or specificity because of the backwards time ordering. I want to know that a predicted risk of 0.2 means that the actual risk is very close to 0.2, and I want decent predictive discrimination as measured by a rank correlation measure or generalized $R^2$. – Frank Harrell May 20 '12 at 3:32
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.