Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I always see in papers that when they want to show how they classifiers performed, they provide ROC graph and the AUC score. Now as far as I know only the AUC tells how well the classifier performed, so what is the advantage of adding the ROC graph? What can one tell from it?

share|improve this question
    
This isn't really any different from lots of cases where the data are described numerically in the text, but people like to present the information visually as well. –  gung Mar 30 at 20:59
add comment

3 Answers 3

The ROC curve is the specificity/sensitivity plot; the AUC is the Area Under Curve. To be brief, the ROC curve can be interesting because it allows comparison of the sensitivity/specificity behaviour of the model. More simply:

$ROC = (x,y) \in R^2 \Rightarrow AUC = z$ but $AUC = z \nRightarrow ROC = (x,y) \in R^2$

share|improve this answer
    
I still don't get it. What could one use the ROC graph for if he only needs the AUC to really evaluate the performance? –  Alex Twain Mar 30 at 18:08
add comment

I usually give the ROC plot but not the AUC: For my applications it is usually clear that either a specific or a sensitive regocognition is needed. The ROC is different for classifiers that are specific but not sensitive vs. sensitive but not specific while the AUC hides this information.

Besides, one can put a whole lot of further information into the plot, e.g. color-coding the thresholds (check whether the classifier is well calibrated if the primary output is posterior probability), model stability (after resampling validation), or confidence regions for a chosen classifier (if one chooses a threshold). Finally, you can even put "extented" measures of sensitivity and specificity which do not require the thresholding @FrankHarrell fights against, e.g. it is possible to extend the concept of sensitivity and specificity and the concept behind Brier's score to yield such measures.

share|improve this answer
    
Is there something about probabilistic decision making that doesn't work in your situation? –  Frank Harrell Mar 31 at 0:11
    
@FrankHarrell: Well, yes: communication. I spend quite a lot of my working time doing translations: physicist <-> chemist (<-> chemometrician) <-> statistician <-> medical doctor, and I find the art of doing this well has a lot to do with allowing simplifications and finding compromises (and trying to avoid to gross misconceptions). I tackle the "don't throw away information" problem by sneaking in continuous measures (as chemist I prefer MSE/RMSE) which are well (or at least better) behaved, but their names and purpose are familiar. –  cbeleites Mar 31 at 14:40
    
I feel that probabilities can work well if communication is careful or the problem involves sports or card playing. –  Frank Harrell Mar 31 at 19:34
add comment

I have not seen a single example where the graph changes our actions or the way we think. I think the ink:information ratio in an ROC graph is enormous. But the worst thing about it is that it tempts us to try to select a cutoff for the predicted risk, which is arbitrary, and inconsistent with optimum decision making.

Beware of the word classifier which implies discarding continuous information.

share|improve this answer
    
Temptation for a cutoff is a point, but OTOH giving the ROC IMHO is already much better than reporting sensitivity and specificity at some arbitrary default cutpoint without any further information. –  cbeleites Mar 30 at 22:54
1  
Only if sensitivity and specificity are helpful. I find them not to be. If we were regressing age on blood pressure we'd never create an ROC curve. Why we do it when $Y$ is binary is beyond me. –  Frank Harrell Mar 30 at 23:34
    
@FrankHarrell regression doesn't help when you need to make a decision, rather than a prediction. At some point you need to define a threshold. You can't treat a patient for 70%. In those settings, ROC curves provide a valuable summary of the model at different operating settings (in part due to the fact that ROC curves are understood by many). That said, I agree on the high ink:information ratio. –  Marc Claesen Mar 31 at 4:57
    
I have to disagree [and medical diagnosis is one of my main areas of research]. There are only two situations I can think of where your reasoning applies: (1) when the analyst is the possessor of the utility function (highly unusual) or (2) when decisions must be made instantly without the time to assess utilities. Not only are probabilities the optimum inputs to true decision makers, they allow one to make no decision, when more data are needed. –  Frank Harrell Mar 31 at 12:18
    
@FrankHarrell: I totally agree that lots of valuable information are lost with the dichotomization. As chemist, my primary philosophical concept of the world is quantitative (substances come at concentrations). So qualitative analysis ("native chemical" version of classification) is the result of comparing a primarily quantitative result to a threshold. However, there are situations which are more basically grouped than continuous. Few people are 20% pregnant... And even in some fundametally continuous situations it just doesn't make sense to cry over the threshold: (...) –  cbeleites Mar 31 at 14:26
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.