Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite
2

I have plot ROC for several models. These models were used to classify my samples into 2 classes.

Using these commands, I can obtain sensitivity vs. specificity plots for each model:

perf <- performance(pred, "sens", "spec")
plot(perf)

Should I rely on the area under the curve (AUC) for each model to conclude which model is better? Other than AUC, should we consider other results so as to conclude which model is better?

If yes, how to get AUC with R? Am I right in assuming that "the smaller it is the better is the classification power of the model?"

share|improve this question
1  
@evdstat You actually hold two unregistered accounts: the present one, and this older one. Please, choose one, register, and we'll merge them afterwards. – chl♦ Jun 27 '11 at 18:38
I don't see what ROC curves, sensitivity, or specificity have to do with the problem at hand. – Frank Harrell Jun 28 '11 at 8:20
@Frank 1. this is a comment to the question, not an answer. 2. AUC, the area under a ROC-Curve is an acceptable way of measuring the predictive power of a classification-model and hence a measure which can be used in the process of model - selection. The ROC itself visualizes the AUC and allows the calculation of a decision threshold. – steffen Jun 28 '11 at 8:29
2  
Right, should have entered this as a comment. The AUC is a useful summary measure once you've finished fitting the model. It is not the ideal objective criterion for selecting the model (likelihood should be used for that). It does not provide a rational basis for a decision threshold as it assumes that utilities are data driven instead of subject driven. – Frank Harrell Jun 28 '11 at 10:34
1  
@Frank I see, maybe we have a term problem. 1. For me as ML, model=anything which predicts the response and modelselection=choose the model whose predictive power is the best (as long as the complexity is not too high etc.) 2. I do not get the point regarding the decision threshold: If one calculates a weighted ROC on basis of a cost-benefit matrix one can determine the optimal decision threshold (bound to the model, of course)... for example for the task of direct marketing (mailing). – steffen Jun 28 '11 at 10:52
show 2 more comments

3 Answers

up vote 4 down vote

AUROC is one of many ways of evaluating the model -- in fact it judges how good ranking (or "sureness" measure) your method may produce. The question whether to use it rather than precision recall or simple accuracy or F-measure is only depending on a particular application.

Model selection is a problematic issue on its own -- generally you should also use the score you believe fits application best, and take care that your selection is significant (usually it is not and some other factors may be important, like even computational time).

About AUC in R -- I see you use ROCR, which makes nice plots but it is also terribly bloated, thus slow and difficult in integration. Try colAUC from caTools package -- it is rocket fast and trivial to use. Oh, and bigger AUC is better.

share|improve this answer
up vote 1 down vote

As, mbq wrote, the answer to whether you should use AUC depends on what you are trying to do. Two points that are worth considering:

AUROC is insensitive to changes in class distribution. It places even emphasis on the different classes, which means it can poorly reflect an algorithm's performance if there is a big imbalance in the distribution of classes. On the other hand, if you are more interested in identifying characteristics of the classes rather than their prevalence, this is a strength.

AUROC does not capture the different costs of different outcomes and it is seldom the case that you care equally about false positives and false negatives.

I find AUROC sensible. The curves easy to read: they are like an intuitive version of a confusion matrix. But it is important to know what we're reading and what's left off.

See also: Evaluating and combining methods based on ROC and PR curves

share|improve this answer
up vote 0 down vote

As you are using ROCR, you can get the point of the ROC curve that maximizes the area and use this to determine the corresponding threshold:

my_prediction <- predict.gbm(object = gbm_mod, newdata = X, 100)
pred <- prediction(my_prediction, Y)
perf <- performance(pred, 'tpr', 'fpr')

r <- rev((as.data.frame(perf@y.values)*(1-as.data.frame(perf@x.values)))[,1])
threshold <- as.data.frame(perf@alpha.values)[which(r==max(r)),1][1]

You can think of this optimization simply as the point that makes the largest possible rectangle under the ROC curve.

share|improve this answer
1  
Although this is correct and helpful in general, frankly, I do not see how this answers any of the OP's questions. Aside, it seems to me that the call to "rev" should be removed. – steffen Nov 6 '12 at 8:54
Oh, I forgot: Welcome to the site :). – steffen Nov 6 '12 at 9:01
Hmm, after re-reading the question, I thought he was asking how to get the optimal threshold for classification, when he was just asking how to use ROC/AUC. Oh well, but you need the rev() because y*(1-x) "reverses" the x axis. – cdgore Nov 7 '12 at 5:43
As far as I see, the OP asked how to select a model based on AUC (and whether this is ok). When determining a threshold solely based on the curve, one can also stick to the AUC. I applied your function on data(ROCR.simple) and found that with and without rev the result is correct, but without rev I get a threshold closer to the optimal point in roc-curve (0,1). – steffen Nov 7 '12 at 9:59
Please see earlier comments. AUC and ROC should play no role. The likelihood function plays a role for a reason, and there are generalized $R^2$ measures based on log likelihood. – Frank Harrell Nov 12 '12 at 20:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.