What is the advantages of the ROC curves?

For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm to select a set of features then I applied SVM for classification. In this case how can I get a ROC curve? Should I change the threshold values of my feature selection algorithm and get sensitivty and specificity of the output to draw a ROC curve?

In my case what is the purpose of creating a ROC curve?

-
"An introduction to ROC analysis" by Tom Fawcett helped me to better understand ROC curves. You might enjoy it if you are looking for additional literature on the topic. –  Alexander May 18 '12 at 16:52
Okay but what would you do to compare two classifiers? If they use thresholds so that niether specificity nor sensitivity match very closely I don't think it is easy to ccompare without lookin at more poitns on the ROC. –  Michael Chernick May 19 '12 at 21:33

## migrated from stackoverflow.comMay 18 '12 at 16:00

This question came from our site for professional and enthusiast programmers.

Many binary classification algorithms compute a sort of classification score (sometimes but not always this is a probability of being in the target state), and they classify based upon whether or not the score is above a certain threshold. Viewing the ROC curve lets you see the tradeoff between sensitivity and specificity for all possible thresholds rather than just the one that was chosen by the modeling technique. Different classification objectives might make one point on the curve more suitable for one task and another more suitable for a different task, so looking at the ROC curve is a way to assess the model independent of the choice of a threshold.

-
Thank you for the reply. It is really useful. Based on the ROC curve is there anyway to determine the threshold? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. –  user570593 May 18 '12 at 16:19
The ROC curve shows you sensitivity and specificity at all possible thresholds, so if you find a point that represents the right tradeoff, you can choose the threshold that goes with that point on the curve. –  Michael McGowan May 18 '12 at 16:21
Is there any automatic way to select the right tradeoff or should I select the tradeoff by myself? And in my case how can I obtain a point in the ROC space for sensitivity=100% or specificity=100%. because I am changing the threshold of the feature selection algorithm. –  user570593 May 18 '12 at 16:23
If you have a well-defined criterion (for instance maximizing precision) then this can be automated. But a good tradeoff for one problem might be lousy for another. –  Michael McGowan May 18 '12 at 16:25
Sensitivity or specificity of 100% can trivially be obtained by setting your threshold at the minimum or maximum value...is that really what you want? –  Michael McGowan May 18 '12 at 16:27
Hi Michael - I'll just add that I like to use proper scoring rules including generalized $R^2$ measures, Brier score (like mean squared error). What I really like to do is to show that predictions are accurate. –  Frank Harrell May 19 '12 at 21:11
@FrankHarrell I'd be interested in what you mean by "99% were not informative"? What type of cases were they and when would using the ROCC be correct? Also how are generalized $R^2$ measures better for binary problems? I am curious because it seems you have quite a strong sentiment against ROCC there and I wonder what's the background. –  Momo May 19 '12 at 23:54
@Momo - I mean that viewing the ROC curve did not help understand model performance very well, and even more the ROC curve did not lead to any insight or to good behavior. Generalized $R^2$ measures are simple translations of likelihood ratio $\chi^2$ statistics and so are powerful. Michael - I'm not really interested in sensitivity or specificity because of the backwards time ordering. I want to know that a predicted risk of 0.2 means that the actual risk is very close to 0.2, and I want decent predictive discrimination as measured by a rank correlation measure or generalized $R^2$. –  Frank Harrell May 20 '12 at 3:32